Source-linked AI summary
When AI Rewrites, Classifiers Relax: Uncertainty-Aware Sentiment Analysis on Sarcastic and AI-Paraphrased Social Text
Shresth Shroff
TL;DR
Standard sentiment evaluations provide limited evidence about classifier behaviour on sarcastic and AI-paraphrased social text. This study measures confidence, accuracy, and abstention across these regimes, finding that AI paraphrases improve accuracy while uncertainty-aware abstention raises retained-set accuracy.
Problem
Standard accuracy and F1 evaluations provide limited evidence about classifier confidence on sarcastic social-media language and behaviour on AI-generated text.
Method
The study evaluates VADER and Twitter-trained RoBERTa across iSarcasm and Yelp Polarity, measuring confidence, AI-paraphrase accuracy, abstention, and uncertainty-signal agreement.
Results
+6.7 pp accuracy from 14% abstention accompanies lower confidence on sarcasm, improved accuracy on AI paraphrases, and near-interchangeable uncertainty signals.
Takeaways & Limitations
Uncertainty-aware abstention is a practical design pattern, while benchmark evaluations should document generative source to avoid overstating performance on organic text.
Takeaways & Limitations
The study covers one review domain and two paraphrase models, uses proxy labels for sarcasm uncertainty, and selects the abstention threshold on the evaluation dataset.
Abstract
from arXiv · showhide
Sentiment classifiers are increasingly applied to social media content that is either sarcastic or AI-generated --- two distributional regimes where standard evaluations offer little guidance. We present a three-part empirical study of sentiment classifier behaviour under these conditions. First, we find that confidence scores on sarcastic text are significantly lower than on non-sarcastic text (Mann--Whitney $p = 2 \times 10^{-6}$), confirming that classifiers sense their own uncertainty on ironic content even without explicit uncertainty modelling. Second, and counterintuitively, we show that sentiment classifiers achieve higher accuracy on AI-paraphrased reviews than on the original human-authored text (RoBERTa: $+5.8$ pp for Qwen3.5-4B paraphrases, $+3.7$ pp for Gemma4-E4B), revealing a cross-domain stylistic alignment effect: AI paraphrases remove distributional noise that confounds Twitter-trained classifiers, producing cleaner, more prototypical sentiment text. Third, we demonstrate that a lightweight abstention wrapper --- flagging the $14\%$ of inputs with confidence below $0.6$ --- improves accuracy from 82.2\% to 88.9\% ($+6.7$ pp) on the retained set. We further compare Semantic Entropy and MC-Dropout-style disagreement as uncertainty signals and find near-identical AUROC ($0.650$ vs.\ $0.646$) on sarcastic text, suggesting that for short social media inputs, both methods are interchangeable. Our results motivate a shift from confident single-label prediction to uncertainty-aware abstention in high-stakes sentiment applications such as mental health flagging and content moderation.
1 Introduction
The introduction argues that standard sentiment benchmarks miss uncertainty and distributional shifts caused by sarcasm and AI-generated text. It presents a unified framework to measure confidence, accuracy changes, abstention, and uncertainty-signal performance, treating AI text as an evaluation confound rather than a detection target.
- Motivation: Standard sentiment evaluation relies on accuracy and F1 on clean, human-annotated benchmarks, failing to capture sarcasm and AI-generated social text.The paper identifies both phenomena as growing challenges for sentiment classifiers trained and evaluated on conventional data.
- Challenge 1: Sarcasm and implicit sentiment: Sarcasm creates a calibration question because positive surface forms can convey negative meaning, challenging models trained on explicit sentiment text.The introduction focuses not only on classification difficulty but also on whether confidence scores reflect that difficulty.
- Empirical framework: The framework evaluates confidence instability, AI-text drift, confidence-threshold abstention, and whether Semantic Entropy or MC-Dropout disagreement better identifies uncertainty on sarcastic text.Experiments use iSarcasm for sarcasm analysis and Yelp Polarity for AI-paraphrase experiments.
- Framing and contribution: Rather than treating AI-generated text as a detection problem, the paper treats it as an evaluation confound that may silently distort downstream classifier metrics.The introduction highlights the counterintuitive finding that AI paraphrases improve rather than degrade classifier accuracy.
2 Background and Related Work
Prior work establishes calibration, sarcasm detection, uncertainty quantification, and human–AI text comparison as relevant foundations, while leaving systematic sentiment confidence analysis on sarcasm and AI-paraphrased text open.
- Sentiment analysis and calibration: Fine-tuned transformers perform well on standard sentiment benchmarks but can be overconfident on out-of-distribution inputs; ECE measures confidence–accuracy gaps, and temperature scaling is standard recalibration.Calibration of sentiment models on ambiguous social media content has not been studied systematically.
- Sarcasm and irony detection: iSarcasmEval established Twitter sarcasm detection with fine-grained labels, while this study examines sarcasm’s effect on sentiment-classifier confidence without explicit sarcasm modeling.Existing work generally treats sarcasm detection as a separate classification task.
- Uncertainty quantification for NLP: Monte Carlo Dropout estimates Bayesian uncertainty through repeated test-time forward passes, whereas Semantic Entropy measures uncertainty over meaning-equivalent generated strings.Prior work also provides context for uncertainty in LLM in-context learning and short-form classification tasks.
- AI-generated text and distributional shift: HC3 enables paired human–ChatGPT text comparisons, while prior studies examine AI authorship attribution and review detectability rather than sentiment accuracy by authorship or paraphrase condition.The paper identifies this sentiment comparison as an unaddressed gap.
3 Experimental Setup
The study evaluates rule-based and neural sentiment baselines on iSarcasm and Yelp Polarity, generating deterministic AI paraphrases and comparing two uncertainty signals under proxy correctness labels. Because iSarcasm lacks sentiment ground truth, sarcasm and uncertainty analyses focus on confidence instability or majority-vote agreement rather than true sentiment accuracy.
- Models evaluated: The evaluation compares VADER, a lexicon-based rule system, with RoBERTa-twitter, a TweetEval-finetuned neural model.Inference runs on CPU for the iSarcasm baseline and on Kaggle T4x2 GPUs.
- Datasets: The sarcasm analysis uses 3,468 iSarcasm tweets, 25% labelled sarcastic, but no sentiment-polarity ground truth.Accordingly, the experiment examines confidence instability rather than sentiment accuracy.
- Datasets: The paraphrase study uses a stratified, balanced 5,000-review Yelp Polarity sample with 2,500 positive and 2,500 negative reviews.Its binary labels make accuracy well-defined.
- AI paraphrase generation: Qwen3.5-4B and Gemma4-E4B generate deterministic paraphrases at temperature = 0.0 while preserving sentiment polarity and key points.The prompt requests different phrasing and only the rewritten text; Qwen3.5 thinking-mode tokens are suppressed.
- Uncertainty comparison setup: The uncertainty comparison samples 300 iSarcasm examples, generates 10 Qwen3.5-4B predictions per example at temperature 0.7, and evaluates Semantic Entropy against MC-Dropout disagreement.AUROC uses a proxy correctness label based on majority-vote reference because iSarcasm lacks sentiment ground truth.
4 Results
Results show that RoBERTa is less confident on sarcastic text but performs better and calibrates more favorably on AI-paraphrased reviews. Confidence-based abstention substantially improves retained-set accuracy, while explanation and uncertainty-signal analyses indicate that flagged cases are genuinely ambiguous and that Semantic Entropy and MC-Dropout disagreement perform similarly.
- Sarcasm and confidence: 0.741 versus 0.770 mean confidence shows that RoBERTa is significantly less confident on sarcastic than non-sarcastic iSarcasm examples.The Mann–Whitney test gives U = 1,005,295 and p = 2 × 10−6.
- AI paraphrasing: AI paraphrases consistently improve RoBERTa accuracy and mostly improve VADER accuracy relative to human-authored Yelp reviews.The proposed mechanism is cross-domain stylistic alignment: paraphrases remove idiosyncratic formatting and idioms, yielding shorter, more prototypical sentiment expressions aligned with RoBERTa-twitter’s training distribution.
- AI paraphrasing: 11.5–17.7% semantic drift means that one in six to one in eight paraphrases receives a different classifier label than the original.For RoBERTa-Qwen, 291 of 573 label changes are net improvements versus 282 net losses.
- Abstention: 14.0% of Yelp examples are flagged below confidence 0.6, raising retained-set accuracy from 82.2% to 88.9% (+6.7 pp).The flagged subset has 41.3% accuracy, while 4,300 of 5,000 examples remain retained.
- Ambiguity explanations: 99.3% of cases flagged by the wrapper contain an ambiguity signal according to Qwen3.5-4B, compared with 83.4% for Gemma4-E4B, and the models agree in 83.0% of cases.Signals include sarcasm, negation, hedging, or conflicting polarity.
- Uncertainty signals: AUROC is approximately 0.65 for both Semantic Entropy and MC-Dropout disagreement, making the methods effectively interchangeable on this task and data regime.Correctness on sarcastic examples uses a proxy label based on model majority vote, which may attenuate both AUROC values.
5 Discussion
The discussion identifies AI paraphrasing as a source of inflated benchmark performance and recommends documenting generative provenance. It also presents abstention as a deployment-ready safeguard while noting limitations in sarcasm labels, paraphrase scope, and threshold selection.
- The stylistic alignment effect: AI-paraphrase augmentation can systematically overestimate sentiment-classifier performance relative to organic social-media text.The authors recommend recording generative source as benchmark metadata alongside domain and annotation method.
- Abstention as a practical design pattern: +6.7 pp accuracy gain from 14% abstention supports uncertainty-aware abstention as a deployment-ready alternative to always-predict systems.The discussion favors human review for ambiguous cases in mental-health triage and content moderation.
- Limitations: Sarcasm analysis is limited to confidence instability rather than accuracy because iSarcasm lacks sentiment-polarity ground truth.The uncertainty AUROC comparison also uses a proxy label.
- Limitations: The AI-paraphrase experiment covers one restaurant/service-review domain and two paraphrase models, limiting generalizability.The abstention threshold of 0.6 was chosen empirically on the evaluation dataset, which may overestimate performance.
6 Future Work
Future work should test whether the stylistic alignment effect generalizes across model architectures and domains, while improving sarcasm data and reviewer support. Proposed extensions include multilingual and code-mixed evaluation and natural-language rationales for flagged inputs.
- 6 Future Work: Future studies should test additional model families, including instruction-tuned Llama-3 and Mistral and multilingual mBERT for code-mixed text.These comparisons would establish whether the stylistic alignment effect generalizes across architectures.
- 6 Future Work: Extending AI-paraphrase analysis to news headlines or social media posts would test whether its effects are domain-specific.
- 6 Future Work: A sarcasm dataset containing both sarcasm labels and sentiment polarity would support further evaluation.
- 6 Future Work: Natural-language rationales for flagged inputs could improve human reviewer efficiency in downstream applications.
7 Conclusion
The study examines sentiment classifier behaviour on sarcastic and AI-paraphrased text, finding lower confidence on sarcasm, improved accuracy on AI paraphrases, and gains from confidence-threshold abstention.
- Conclusion: The study evaluates sentiment classifiers on two underexplored regimes: sarcastic text and AI-paraphrased text.These regimes frame the paper’s structured empirical analysis.
- Conclusion: Classifiers display measurably lower confidence on sarcastic inputs without explicit uncertainty training.This indicates that confidence responds to sarcastic input conditions even without dedicated uncertainty modelling.
- Conclusion: AI paraphrases counterintuitively improve classification accuracy through a cross-domain stylistic alignment effect.The conclusion attributes the accuracy improvement to stylistic alignment across domains.
- Conclusion: 6.7 percentage points of accuracy are recovered by a simple confidence-threshold abstention wrapper.The wrapper uses confidence thresholding to abstain on uncertain inputs.