PaperlayerView exact revision on arXiv

Source-linked AI summary

SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Filipe Nunes Ribeiro, Matheus Araújo, Pollyanna Gonçalves, Fabrício Benevenuto, Marcos André Gonçalves

arXiv:1512.01818v5cs.CLcs.SI

TL;DR

Sentiment-analysis methods lack thorough, practice-oriented comparisons across diverse datasets, leaving their relative performance unclear. SentiBench evaluates 24 methods on 18 labeled datasets and finds that performance varies substantially by dataset, with no method consistently best.

  • Problem

    The relative performance, advantages, and limitations of sentiment-analysis methods remain insufficiently established across datasets and contexts.

  • Method

    The paper benchmarks 24 sentence-level sentiment-analysis methods on 18 labeled datasets spanning social networks, reviews, news, talks, and blogs.

  • Results

    Prediction performance varies considerably across datasets, and no single method consistently achieves the best performance across them.

  • Takeaways & Limitations

    Sentiment-analysis methods should not be treated as off-the-shelf tools, so researchers and companies should test different methods before applying one.

  • Takeaways & Limitations

    Even the best evaluated methods leave substantial room for improvement, with Macro-F1 around 0.6 in three-class experiments.

Abstract

from arXiv · show

In the last few years thousands of scientific papers have investigated sentiment analysis, several startups that measure opinions on real data have emerged and a number of innovative products related to this theme have been developed. There are multiple methods for measuring sentiments, including lexical-based and supervised machine learning methods. Despite the vast interest on the theme and wide popularity of some methods, it is unclear which one is better for identifying the polarity (i.e., positive or negative) of a message. Accordingly, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, \textit{as they are used in practice}, across multiple datasets originated from different data sources. Such a comparison is key for understanding the potential limitations, advantages, and disadvantages of popular methods. This article aims at filling this gap by presenting a benchmark comparison of twenty-four popular sentiment analysis methods (which we call the state-of-the-practice methods). Our evaluation is based on a benchmark of eighteen labeled datasets, covering messages posted on social networks, movie and product reviews, as well as opinions and comments in news articles. Our results highlight the extent to which the prediction performance of these methods varies considerably across datasets. Aiming at boosting the development of this research area, we open the methods' codes and datasets used in this article, deploying them in a benchmark system, which provides an open API for accessing and comparing sentence-level sentiment analysis methods.

Introduction

The introduction identifies a lack of established relative performance among widely used sentiment-analysis methods and motivates an apple-to-apple benchmark across datasets and data sources. The article compares twenty-four state-of-the-practice methods on eighteen labeled datasets and releases the methods, data, and a comparison system.

  • Research gap: Existing sentiment-analysis methods are rarely compared comprehensively across many datasets, leaving their relative performance poorly understood.The state of the art is not clearly established, and researchers often accept popular methods without understanding their comparative advantages, disadvantages, and limitations.
  • Motivation: The study calls for a thorough apple-to-apple comparison of methods as used in practice across datasets from different data sources.This need follows from widespread deployment without a deeper understanding of methods’ applicability across contexts.
  • Study design: Twenty-four state-of-the-practice methods are benchmarked using eighteen labeled datasets.The benchmark covers datasets from social networks, movie and product reviews, and opinions or comments in news articles.
  • Main findings: No single method consistently achieves the best prediction performance across all datasets.The methods also vary widely in agreement, including across similar datasets, so the same content may receive different interpretations depending on the chosen method.
  • Resources: The authors release the gold-standard dataset and compared-method codes and provide a Web system for testing methods on particular datasets or applications.The system enables researchers and practitioners to use the released data and codes to compare results with existing methods.

Background and Related Work · Focus on Sentence-Level Sentiment Analysis · Existing Efforts on Comparison of Methods

The paper narrows sentiment analysis to sentence-level polarity detection and evaluates methods usable without manual labeling. It situates this benchmark against prior comparisons, which generally used limited datasets or restricted participation.

  • Focus on Sentence-Level Sentiment Analysis: The benchmark focuses on detecting positivity or negativity in short texts, a common function across the sentiment methods considered.This task is especially relevant to applications analyzing short social-media messages.
  • Focus on Sentence-Level Sentiment Analysis: Sentence-level sentiment analysis includes supervised methods requiring labeled training data and lexical methods using predefined sentiment-associated word lists.Supervised approaches can adapt to specific contexts, whereas lexical approaches rely on predefined vocabularies.
  • Focus on Sentence-Level Sentiment Analysis: The study evaluates unsupervised-use methods because they can be deployed without human labeling or other manual intervention.Some included tools used machine learning internally but were released as black-box systems usable in an unsupervised manner.
  • Existing Efforts on Comparison of Methods: Only a limited number of existing methods comparisons covered sentiment analysis, and those comparisons usually relied on restricted datasets.Lexical and machine-learning approaches had evolved in parallel, motivating comparisons that used one strategy as a baseline for the other.
  • Existing Efforts on Comparison of Methods: SemEval compared competitors through sentiment-analysis tracks, but registration requirements excluded many popular methods from evaluation.Its labeled datasets also typically focused on one data type, such as tweets, rather than a broad range of social-media data.
  • Existing Efforts on Comparison of Methods: Ahmadi et al. compared twenty tools across five Twitter datasets, whereas this study covers distinct contexts including reviews, comments, and social networks.The cited prior benchmark included both supervised and unsupervised approaches, while the present study emphasizes broader contextual coverage.
  • Existing Efforts on Comparison of Methods: A previous study compared eight sentence-level methods on one public SentiStrength dataset; this article expands the comparison across more methods and datasets.The methods in this paper were also incorporated into the iFeel system.

Sentiment Analysis Methods · Adapting Lexicons for the Sentence Level Task

The paper reviews twenty-four sentence-level sentiment methods and adapts lexicons with VADER’s sentence-processing heuristics. These heuristics substantially improve most lexicon results over averaging positive and negative word occurrences.

  • Sentiment Analysis Methods: Twenty-four sentence-level sentiment analysis methods are systematically identified from major field conferences, cited papers, references, and author-shared or publicly downloadable implementations.
  • Sentiment Analysis Methods: Tables 1 and 2 summarize each method’s lexicon or machine-learning techniques, outputs, validation datasets, comparison baselines, and lexicon sizes.
  • Sentiment Analysis Methods: The methods are organized chronologically, and they produce different output formats representing negative, neutral, and positive sentiment.
  • Adapting Lexicons for the Sentence Level Task: A lexicon alone cannot naturally infer sentence polarity, so sentence-level methods must receive sentences as input and produce polarities as output.
  • Adapting Lexicons for the Sentence Level Task: VADER and SO-CAL combine lexicons with sentence-characteristic processing, including intensifiers, punctuation transformations, emoticons, and other heuristics.
  • Adapting Lexicons for the Sentence Level Task: VADER’s implementation is used to evaluate lexicons because it is open source for replication and had shown expressive results in previous experiments.
  • Adapting Lexicons for the Sentence Level Task: VADER’s heuristics address punctuation, capitalization, degree modifiers, “but” conjunctions, and trigram-based negation to capture sentiment-intensity changes beyond bag-of-words.
  • Adapting Lexicons for the Sentence Level Task: The VADER heuristics drastically improve most evaluated lexicons over a baseline that averages positive and negative word occurrences to classify sentence polarity.

SentiWordNet. · Output Adaptations

The benchmark adapted heterogeneous sentiment-method outputs into comparable positive, negative, or neutral sentence-level predictions. These adaptations included mapping lexicon-specific affect categories, applying VADER heuristics, aggregating clue polarities, and thresholding or interpreting existing outputs.

  • Output Adaptations: PANAS-t’s eleven moods were mapped into positive, negative, or neutral affect categories for comparison.Joviality, assurance, serenity, and surprise were positive; fear, sadness, guilt, hostility, shyness, and fatigue were negative; attentiveness was neutral.
  • Output Adaptations: Emolex’s eight sentiment associations were adapted using additional positivity and characterization entries from its authors.The passage specifies that Emolex provides associations with eight sentiments and that two additional released entries were considered.
  • Output Adaptations: SentiWordNet and other lexicons with positive-negative scores were processed with VADER heuristics to produce VADER-style scores.This placed their outputs into the same scoring framework used for VADER’s output.
  • Output Adaptations: OpinionFinder’s sentence polarity was computed by summing the -1, 0, or 1 polarities of all sentiment clues in the sentence.The available implementation can produce multiple clues for one sentence, so their polarities were aggregated.
  • Output Adaptations: SO-CAL and Pattern.en outputs were classified by thresholding float values into positive or negative categories.Values greater than the threshold indicated positive sentiment, while values below the threshold indicated negative sentiment.
  • Output Adaptations: LIWC, SenticNet, SASA, USent, SentiStrength, Umigon, VADER, and Semantria already provided fixed outputs for the three target classes.These methods directly indicated positive, negative, or neutral outputs, while Stanford Recursive Deep Model produced very negative and very positive outputs.

Paid Softwares · Methods not included

The benchmark included seven closed paid software methods, with specified academic or author-provided access arrangements for some tools. Several highly cited methods were excluded because they were unavailable or could not be reimplemented from published descriptions.

  • Paid Softwares: Seven of the twenty-four evaluated methods were closed paid software tools.The paid methods were LIWC (2007 and 2015), Semantria, SenticNet 3.0, Sentiment140, and SentiStrength.
  • Paid Softwares: The paid-software group included LIWC (2007 and 2015), Semantria, SenticNet 3.0, Sentiment140, and SentiStrength.
  • Paid Softwares: SentiStrength had a free-of-charge academic license despite being paid commercially.
  • Paid Softwares: SenticNet’s authors processed all datasets with the commercial version and returned the resulting polarities.
  • Paid Softwares: The evaluation used SentiStrength’s Java version from May 2013, packaged with all commercial features.
  • Methods not included: POMS-ex and Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification were excluded because they were unavailable and could not be reimplemented from their descriptions.
  • Methods not included: NRC SVM was also unavailable for inclusion in the comparison.

Datasets and Comparison Among Methods

Existing sentiment-analysis methods use highly varied validation strategies, datasets, and baselines, ranging from toy examples to large labeled datasets. Because no standard benchmark exists, comparisons across methods remain inconsistent and fragmented across research communities.

  • Validation and comparison practices: Validation strategies, datasets, and baseline comparisons vary greatly, spanning toy examples to large labeled datasets.PANAS-t and Emoticons DS used manually unlabeled Twitter data, while other methods used different validation settings.
  • Validation and comparison practices: Lexicon studies often validate dictionaries against other dictionaries or human labels without comparing the lexicon as a standalone sentiment-analysis method.AFINN compared dictionaries, Emoticon Distance Supervised used Pearson correlation with human labeling, and SentiWordNet used dictionary and human validation.
  • Need for a benchmark: No standard benchmark exists because methods emerge across research communities using different techniques and limited awareness of related efforts.This lack of standardization produces comparisons based on different solutions and datasets.

Gold Standard Data

The benchmark uses eighteen labeled datasets assembled mainly through non-expert human annotation, with majority voting commonly assigning sentence polarity. A small expert review assessed dataset trustworthiness, finding disagreements reasonable given mixed-polarity interpretations and moderate-to-substantial agreement ranges.

  • Dataset construction: Eighteen datasets were characterized by message counts, average message length, and classification methodology, with human labeling used in almost all datasets.Reviews I instead relied on five-star user ratings.
  • Dataset construction: Seven of eighteen datasets used Amazon Mechanical Turk, while ten used volunteers or other non-expert evaluators.Majority voting usually produced the final agreed-upon polarity for each sentence.
  • Validation: Experts evaluated a random 1% sample of all sentences to assess agreement with the existing gold-standard labels.The procedure was intended as a trustworthiness check rather than a full repetition of the original human evaluations.
  • Validation: Cohen’s Kappa measured agreement between the authors’ evaluations and each dataset’s gold-standard labels.Disagreements sometimes reflected plausible alternative interpretations, including mixed-polarity sentences.
  • Validation: Kappa values between 0.4 and 0.6 indicate moderate agreement, while values amid 0.60 and 0.8 correspond to substantial agreement.These ranges support treating the observed disagreements as reasonable and expected.

Comparison Results

The section compares twenty-four methods using eighteen gold standard datasets.

  • Twenty-four methods are included in the comparison.
  • The evaluation uses eighteen gold standard datasets.
  • The section presents comparison results across the twenty-four methods and eighteen datasets.

Experimental details

The experiments compare sentiment methods under both three-class and two-class settings. The two-class evaluation excludes neutral sentences and uses coverage to assess methods that fail to assign positive or negative polarity.

  • Method approaches: Sentiment analysis approaches differ in whether they first identify objective or neutral sentences and then classify polarity, or directly use three positive, negative, and neutral classes.OpinionFinder and Pattern.en exemplify the two-step approach.
  • Experimental design: The study performs two rounds of tests: three-class prediction of positive, negative, and neutral, followed by two-class prediction of positive versus negative.The three-class experiments use only datasets containing considerable numbers of neutral messages, excluding Twitter datasets.
  • Experimental design: In the two-class experiments, all datasets are used after excluding neutral sentences, and all methods—including those producing neutral outputs—are evaluated.Sentences with undefined polarity are treated as neutral for two-class methods.
  • Evaluation metric: Coverage is introduced to measure how methods handle sentences whose polarity cannot be identified as positive or negative.Two-class methods commonly assign such sentences an undefined polarity, even though the evaluated sentences are positive or negative.

Comparison Metrics

The comparison uses class-sensitive and aggregate metrics for two- and three-class experiments, supplemented by mean rankings and Friedman’s statistical test. Macro-F1 underpins rankings and complements accuracy, while Friedman’s test evaluates whether method performance differs across datasets.

  • Classification Metrics: Three-class experiments use Precision, Recall, and F1 to evaluate automated classification.Precision measures correct predictions among predicted class members, Recall measures correct classifications among known class members, and F1 is their harmonic mean.
  • Aggregate Metrics: Accuracy counts correct classifications equally across sentences, whereas Macro-F1 averages class-specific F1 values and gives each class equal importance.Macro-F1 is intended for skewed datasets, so accuracy and Macro-F1 provide complementary assessments.
  • Two-Class Experiments: The same Precision, Recall, and F1 metrics apply to two-class experiments after removing the neutral rows and columns.For the positive class, precision is a/(a+c), recall is a/(a+b), and F1 is 2P(pos)·R(pos).
  • Overall Ranking: Mean Ranking averages a method’s dataset-specific ranks, with ranks calculated using Macro-F1.It is proposed as a simple measure for summarizing performance across many method, metric, and dataset combinations.
  • Statistical Testing: Friedman’s Test assesses whether methods show globally similar performance across datasets by testing rankings sorted by decreasing Macro-F1.With more datasets, the test uses a chi-square approximation with c−1 degrees of freedom; at α = 0.05, rejecting the null indicates significant rank differences.

Comparing Prediction Performance

Prediction performance varies substantially across datasets and tasks, with no universally consistent best method. The results therefore support context-specific method selection, preliminary evaluation on new datasets, and continued improvement of existing methods.

  • Cross-dataset variability: Performance varies considerably across datasets, and the best method for each dataset also changes substantially.The authors report significant differences in mean rankings across datasets, indicating that no method consistently occupies the same rank.
  • Performance limitations: The strongest methods still have limited overall performance, with Macro-F1 around 0.9 only for low-coverage methods in 2-class experiments and only 0.6 in 3-class experiments.The benchmark identifies substantial room for improvement despite acceptable performance from the evaluated methods.
  • Class-specific performance: Twelve of twenty-four methods classify positive messages more accurately than negative or neutral messages, while neutral messages are harder for most methods to detect.The authors suggest that some methods may be biased toward positivity, consistent with a broader positivity bias in human language.
  • Coverage and context: For 2-class experiments, mean-rank comparisons must be interpreted alongside coverage because some methods identify polarity for only a small portion of input text.In social-network datasets, SentiStrength was the winner with coverage around 30%, while Emoticons and Panas-t had about 18% and 6% coverage, respectively.
  • Overall rankings: VADER ranks best overall in 3-class experiments despite never ranking first on an individual dataset, showing consistency across datasets rather than repeated dataset-level wins.VADER ranked second five times, third twice, seventh three times, and fourth, sixth, and fifth once each.
  • Transfer to new datasets: Methods often perform better on datasets used for their original validation, so preliminary testing is crucial before applying a method to a new dataset.When prior testing is infeasible, the benchmark provides comparative information for researchers and companies selecting sentiment-analysis methods.

Concluding Remarks

The paper concludes that sentiment-method performance varies substantially across datasets, leaving considerable room for improvement and motivating comparative experimentation. It also contributes openly released datasets, code, and the iFeel web service for comparing methods.

  • Benchmark contribution: Twenty-four popular sentence-level sentiment analysis methods were compared using gold-standard data.The methods originated in linguistics and psychology and were applied to quantify moods in unstructured short online-social-network messages.
  • Findings: Prediction performance varies largely across datasets, and overall results leave substantial room for improvement.A few methods appear among the best on different datasets, but no method consistently resolves the cross-dataset variability.
  • Practical implication: The same social-media text can receive very different sentiment interpretations depending on the selected method.The authors therefore recommend that researchers and companies experiment with different methods before applying one.
  • Open resources: The authors released the paper’s datasets and code and incorporated them into the iFeel web service for comparing sentiment-analysis methods.They intend this resource to support researchers and practitioners and encourage further research using a rigorous scientific approach.
Loading 1512.01818v5…