Source-linked AI summary
Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts
Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson, Michele C. Weigle
TL;DR
Because sentiment tools may be used without their limitations being understood, this study compares three bespoke tools and three LLMs with six human raters on 100 tweets using Cohen’s and Fleiss’ kappa. Agreement was only fair even among humans; binary labels agreed more than three-class labels, and Twitter-RoBERTa-base aligned most strongly with human judgments.
Problem
Sentiment tools are often applied without their limitations being understood, although short, context-poor social-media texts produce inconsistent human sentiment judgments.
Source: §1 · p. 2Method
The study compared three bespoke sentiment tools and three LLMs against six human raters on 100 tweets using Cohen’s kappa for pairs and Fleiss’ kappa for multiple raters.
Source: §1 · p. 2, §3.2 · p. 4Results
Binary classifications yielded higher agreement than three-class labels; Twitter-RoBERTa-base aligned best with humans, while LLMs aligned better on positive vs. non-positive classifications.
Source: §6 · pp. 8–9Takeaways & Limitations
Human-verified labels and multi-rater evaluation remain important for establishing a reliable gold standard, and automated tools should not generate ground truth.
Source: §6 · p. 9The evaluation covers 100 tweets from one topic and location; broader applicability needs testing across additional topics, geographic regions, and social-media communities.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limitations. In this study, we evaluate the inter-rater reliability of three bespoke sentiment analysis tools (TextBlob, VADER, and Twitter-roBERTa-base) and three large language models (LLMs: Qwen3-32B, GPT-OSS-120B, Llama-4-Maverick-17B) against six human raters across 100 tweets. We measured agreement using two statistical measures: Cohen's kappa for pairwise comparisons and Fleiss' kappa for multiple raters. Even among the human raters, our results showed only fair agreement, highlighting the subjectivity of sentiment analysis. Higher agreement was observed under the binary sentiment classification (negative vs. non-negative and positive vs. non-positive) than under the three-class classification across both humans and automated tools. The Twitter-roBERTa-base model showed the strongest alignment with human ratings, outperforming both bespoke sentiment tools and LLMs, particularly in distinguishing negative versus non-negative sentiment. LLMs showed substantial agreement among themselves and moderate to substantial alignment with humans, performing better in positive vs. non-positive classifications. Our findings underscore that domain-specific fine-tuning remains crucial for reliable social media sentiment analysis, and human-centered evaluation remains essential for establishing gold-standard labels.
1 Introduction
The introduction motivates careful evaluation of social-media sentiment tools and compares automated systems with human judgments. It reports fair agreement even among humans, stronger agreement for binary labels, and the strongest human alignment from Twitter-RoBERTa-base.
1 Introduction
- Sentiment tools are often used without fully understanding their limitations, and accepting their outputs at face value can mislead when human interpretation is nuanced.The authors report that humans themselves do not consistently agree on short, context-poor posts.Source: §1 · p. 2
- The study compared three bespoke sentiment tools and three LLMs against six human raters on 100 tweets using Cohen’s and Fleiss’ kappa.The small dataset allowed detailed qualitative review and independent annotation by all six raters.Source: §1 · p. 2
- Twitter-RoBERTa-base showed the best alignment with human ratings, while binary classifications agreed more than three-class classifications, especially negative vs. non-negative.The study also found that even human raters only showed fair agreement with one another.Source: §1 · p. 2
- The paper argues that automated tools and LLMs should be used cautiously and that human-verified labels remain the benchmark for evaluating sentiment.The authors recommend multi-rater evaluation for establishing a reliable gold standard.Source: §1 · p. 2
2 Background and Related Work
The background describes sentiment analysis in social media and reviews established tools and models. The study’s distinctive approach is a fine-grained comparison with independent human judgments rather than reliance on pre-labeled datasets or large-scale automated labeling.
2 Background and Related Work
- Social-media posts, particularly on Twitter, are brief and informal and often contain slang, hashtags, and sarcasm.The background identifies social-media sentiment analysis as a widely used application area.Source: §2 · pp. 2–3
- The study tested TextBlob, VADER, and Twitter-RoBERTa alongside Qwen3-32B, GPT-OSS-120B, and Llama-4-Maverick-17B using zero- and one-shot prompting.One-shot results were reported because they performed slightly better or comparably, while zero-shot results did not affect the overall conclusions.Source: §2 · p. 3
- Prior comparisons report that VADER often outperforms TextBlob on short, casual posts, while transformer models capture nuanced, context-dependent sentiment.TextBlob and VADER remain widely used for their simplicity and accessibility.Source: §2 · p. 3
- Unlike studies relying on pre-labeled datasets or large-scale automated labeling, this study manually evaluated 100 tweets with three tools and six independent human raters.The design enabled direct comparison with human judgments and examination of their diversity and subjectivity.Source: §2 · p. 3
3 Methodology
The methodology evaluates automated sentiment labels against six human annotations of a curated 100-tweet dataset. Pairwise and multi-rater kappa measures assess agreement across three-class and binary sentiment classifications.
3 Methodology
- The dataset comprised 100 “Site C, Khayelitsha” tweets posted from January 2022 to December 2024, labeled by three tools, three LLMs, and six human raters.The six raters were Computer Science graduate students from the authors’ lab with substantial exposure to social media and sentiment analysis.Source: §3 · p. 4
- The analysis compared each pair of human raters and tools, then used human majority-vote labels to evaluate tool performance; no majority-vote ties occurred.The evaluation combined Cohen’s kappa for pairs with Fleiss’ kappa across multiple raters.Source: §3 · p. 4
3.1 Measuring Inter-Rater Agreement: Cohen’s Kappa
- Cohen’s kappa measured pairwise categorical agreement, and the study tested whether negative/non-negative or positive/non-positive labels improved agreement over three classes.The coefficient ranges from -1 to 1; the authors interpret values from 0.21–0.40 as fair and 0.81–1.00 as almost perfect agreement.Source: §3.1 · p. 4
3.2 Multi-Rater Agreement: Fleiss’ Kappa
- Fleiss’ kappa generalized agreement assessment to multiple raters across three-class and two binary sentiment tasks; Fig. 1 displays the pairwise Cohen’s kappa comparisons.In the figure, panels show positive/negative/neutral, negative/non-negative, and positive/non-positive classifications, with darker values indicating better agreement.Source: §3.2 · p. 4, Figure 1 · p. 5
4 Results
The domain-specific Twitter-RoBERTa model aligned most closely with the human majority overall, while LLMs aligned more strongly on positive/non-positive classification. Agreement was generally higher for binary than three-class sentiment, though human judgments also varied.
4 Results
- Cohen’s kappa for sentiment analysis tools (TB, VD, RB) generally indicated slight-to-fair agreement with human raters R1–R6; RB’s highest human pairing was 0.72 with R1.LLMs generally achieved fair-to-moderate agreement with human raters R1–R6; GP’s highest human pairing was 0.74 with R6, and LM–GP agreement was 0.80.Source: §4 · pp. 5–6
4.1 Higher Inter-Rater Agreement Under Binary Sentiment Classification
- Binary classifications generally produced higher agreement than three-class classification, with negative/non-negative agreement strongest for humans and non-LLM tools.LLMs tended to align more with humans on positive sentiment.Source: §4.1 · p. 6
4.2 Human Consensus via Majority Voting Improves Agreement with Automated Tools
- Human raters varied in their interpretations, and majority voting was used to form a more stable consensus for comparison with automated systems.The paper reports that majority voting improved consistency and alignment with automated tools.Source: §4.2 · pp. 6–7, Table 1 · p. 7
- RB had the highest agreement with the human majority overall, except in positive/non-positive classification, where LLaMA was highest.RB’s Cohen’s kappa was 0.88 for negative/non-negative, 0.73 for positive/negative/neutral, and 0.61 for positive/non-positive.Source: Table 2 · p. 7, §4.2 · pp. 7–8
5 Limitations and Future Work
The study’s findings are bounded by a curated sample of 100 tweets, categorical labels, and agreement estimates reported without confidence intervals or significance tests.
5 Limitations and Future Work
- The evaluation covered 100 curated tweets, so broader applicability across topics, regions, and social media communities remains to be assessed.The authors propose expanding the approach to those settings.Source: §5 · p. 8
- Converting continuous polarity scores and class probabilities into discrete labels discarded information about sentiment intensity and prediction uncertainty.The authors suggest continuous or probability-based comparisons for future work.Source: §5 · p. 8
- Agreement statistics are point estimates without confidence intervals or formal significance tests, so differences should be interpreted descriptively.The authors suggest larger datasets and confidence intervals or hypothesis tests to assess statistical reliability.Source: §5 · p. 8
6 Conclusions
The study finds that sentiment agreement varies across humans and automated systems, with stronger agreement in binary tasks than in three-class classification. Twitter-RoBERTa aligns most closely with human ratings, while human-verified evaluation remains central to establishing reference labels.
6 Conclusions
- Binary sentiment classifications had higher agreement than three-class classification, with negative/non-negative classification showing the strongest agreement for tools and humans.LLMs aligned more strongly with humans on positive/non-positive classification.Source: §6 · pp. 8–9
- Twitter-RoBERTa showed the strongest alignment with human ratings, including substantial or almost perfect agreement across sentiment categories.The authors connect this result to the potential of carefully trained, domain-specific models to outperform general-purpose LLMs.Source: §6 · p. 9
- LLMs agreed substantially among themselves and moderately to substantially with the human majority, but did not automatically outperform classical sentiment tools on social media text.LLMs aligned better on positive/non-positive classification.Source: §6 · p. 9
- TextBlob and VADER showed weaker agreement than other evaluated systems, and TextBlob had the lowest agreement across categories.The conclusion notes that both tools struggled particularly with positive/non-positive sentiment.Source: §6 · p. 9
- Because human raters also disagreed, the authors identify multi-rater evaluation and human-verified labels as essential for establishing a reliable sentiment gold standard.They caution against using automated tools or LLMs to generate ground truth.Source: §6 · p. 9