PaperlayerView exact revision on arXiv

Source-linked AI summary

Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best-Worst Scaling

Svetlana Kiritchenko, Saif M. Mohammad

arXiv:1712.01741v1cs.CL

TL;DR

Fine-grained sentiment scores are difficult to annotate consistently, despite their usefulness for sentiment analysis and related applications. The paper uses Best–Worst Scaling to score words and phrases across general English, English Twitter, and Arabic Twitter. Rankings remain remarkably consistent across repeated annotation, and the paper estimates the least perceptible sentiment difference for English and Arabic speakers.

  • Problem

    Manual sentiment lexicons often use coarse positive/negative categories, while fine-grained real-valued scores are challenging to annotate consistently.

    Source: §1 · pp. 1–2

  • Method

    The paper applies Best–Worst Scaling to create sentiment composition lexicons with real-valued scores for words and phrases in three domains.

    Source: §1 · pp. 1–2, §2.1 · p. 2

  • Results

    Repeated annotations yield similar term rankings across all three domains, with an average Spearman rank correlation of 0.98; the least perceptible difference is about 0.08 for English and Arabic speakers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported reliability supports estimating the least perceptible sentiment difference, which can help researchers study sentiment composition.

    Source: §1 · p. 2, §4 · p. 5

Abstract

from arXiv · show

Access to word-sentiment associations is useful for many applications, including sentiment analysis, stance detection, and linguistic analysis. However, manually assigning fine-grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annotation technique of Best-Worst Scaling to obtain real-valued sentiment association scores for words and phrases in three different domains: general English, English Twitter, and Arabic Twitter. We show that on all three domains the ranking of words by sentiment remains remarkably consistent even when the annotation process is repeated with a different set of annotators. We also, for the first time, determine the minimum difference in sentiment association that is perceptible to native speakers of a language.

1 Introduction

The paper addresses the difficulty of obtaining consistent fine-grained sentiment scores and applies Best–Worst Scaling to create sentiment composition lexicons across three domains. It reports reliable rankings across repeated annotations and estimates the least perceptible score difference for English and Arabic speakers.

1 Introduction

  • Existing manual lexicons often use coarse positive/negative categories, while real-valued annotation raises cognitive-load and annotator-consistency challenges.Directly comparing pairs can require order N^2 annotations; Best–Worst Scaling is presented as a comparative approach that keeps annotation counts small.Source: §1 · pp. 1–2
  • The paper creates fine-grained sentiment composition lexicons for general English, English Twitter, and Arabic Twitter, with real-valued scores for single words and multi-word phrases.The English Twitter lexicon builds on earlier work; the general English and Arabic Twitter lexicons are novel contributions in this paper.Source: §1 · pp. 1–2
  • An average Spearman rank correlation of 0.98 shows that annotations across all three domains produce similar term rankings when repeated with different annotators.Reliable rankings can be obtained with two or three annotations per BWS question, compared with 15 to 20 responses for conventional rating-scale methods.Source: §1 · p. 2
  • About 0.4 sentiment-score difference corresponds to 90% agreement when annotators choose which of two terms is more positive.This is 20% of the interval between -1 and 1.Source: §1 · p. 2
  • About 0.08 is the least perceptible sentiment-score difference for English and Arabic speakers on the scale from -1 to 1.The paper suggests using this threshold to assess whether a phrase’s sentiment differs meaningfully from that of its constituent noun.Source: §1 · p. 2

2 Capturing Fine-Grained Sentiment Associations By Manual Annotation

The paper uses manual annotation to produce real-valued sentiment association scores for words and phrases. Best–Worst Scaling asks annotators to select the most and least positive items from four-item sets, and the resulting lexicons cover three English and Arabic domains.

2.1 Best–Worst Scaling Method of Annotation

  • Best–Worst Scaling (BWS), also called Maximum Difference Scaling, asks annotators to select the most positive and most negative items in a four-item set, yielding real-valued association scores and a ranking.Each term’s score is the percentage of times chosen most positive minus the percentage chosen most negative, ranging from -1 to 1.Source: §2.1 · pp. 2–3
  • Terms are sampled into four-term sets with each term and pair appearing approximately equally often; around 1.5n to 2n BWS questions, where n is the number of items, suffice for reliable scores.The three lexicons were annotated separately, with 2n four-tuples generated for each set.Source: §2.1 · p. 3
  • CrowdFlower annotators judged four terms at a time, choosing the most and least positive; Arabic used a similar questionnaire in Arabic.The responses were converted into scores and term rankings through a counting procedure.Source: §2.1 · p. 3

2.2 Lexicons Created With Best–Worst Scaling

  • The three lexicons cover English Twitter, Arabic Twitter, and general English, including single words and phrases such as negations, modals, and degree adverbs.The English Twitter lexicon contains 1,515 terms, the Arabic Twitter lexicon 1,367 terms, and the general English lexicon combines 1,621 words with 1,586 modifier phrases.Source: §2.2 · p. 3

3 Quality of Annotations

Across the three lexicons, BWS annotations yield highly reproducible rankings, and sentiment-score differences track human judgments. The section also describes score coverage and estimates the sentiment differences native speakers can perceive.

3.1 Agreement and Reproducibility

  • Majority answers matched 82% of English Twitter responses, 80% of Arabic Twitter responses, and 80% of General English responses.Source: §3.1 · p. 3
  • Across three lexicons, split sets correlated at least 0.98 by Spearman; two- and three-annotation subsets correlated 0.98 and 0.99 with ten, respectively.Each term occurs in eight 4-tuples on average, so one annotation per question still assesses each term eight times.Source: §3.1 · p. 4

3.2 Distribution of Sentiment Scores

  • SCL-NMA scores cover the full -1 to 1 range without significant gaps, though the lexicon has slightly fewer terms at the extremes.Figure 1 plots each term’s rank against its BWS sentiment score.Source: Figure 1 · p. 4, §3.2 · p. 4

3.3 Perception of Sentiment Difference

  • At score differences of 0.4 or higher, more than 90% of annotators identify the more-positive term, and agreement rises rapidly as differences increase.Figure 2 plots average human agreement against the score difference.Source: Figure 2 · p. 5, §3.3 · p. 5
  • The least perceptible differences were 0.069 for SCL-NMA, 0.080 for the English Twitter Lexicon, and 0.087 for the Arabic Twitter Lexicon.The measure is the point where the 99.9%-confidence lower bound on agreement consistently exceeds the 50% chance threshold.Source: §3.3 · p. 5

4 Conclusions

Best–Worst Scaling produced real-valued sentiment association scores for single words and multi-word phrases across three domains. Repeating the procedure yielded remarkably consistent sentiment rankings, enabling measurement of the least perceptible difference in sentiment.

  • Repeated Best–Worst Scaling produced remarkably consistent rankings of terms by sentiment.Source: §4 · p. 5
  • The lexicons cover single words and multi-word phrases in general English, English Twitter, and Arabic Twitter.Source: §4 · p. 5
  • Best–Worst Scaling uses comparative annotation while keeping the number of annotations small.The observed reliability allowed the authors to determine the least perceptible difference in sentiment.Source: §4 · p. 5
Loading 1712.01741v1…