Source-linked AI summary
A Large Self-Annotated Corpus for Sarcasm
Mikhail Khodak, Nikunj Saunshi, Kiran Vodrahalli
TL;DR
Sarcasm detection lacks large, naturally distributed, context-rich corpora with author-provided labels. The paper introduces SARC, a self-annotated Reddit corpus with benchmarks and baseline evaluations, containing over 1 million sarcastic sentences and many more non-sarcastic statements. The authors conclude that the corpus and benchmarks provide a foundation for improving sarcasm detection, while annotation noise remains a limitation.
Problem
Sarcasm detection needs corpora beyond balanced or human-annotated datasets, particularly large datasets with naturally occurring labels and contextual information.
Method
The paper constructs SARC from self-annotated Reddit comments, preserves author, topic, and conversation context, and derives benchmark tasks evaluated with baseline and human methods.
Results
Over 1 million sarcastic sentences are provided, alongside many more non-sarcastic statements, with baseline methods evaluated against human performance.
Takeaways & Limitations
SARC offers freely available raw data and evaluation subsamples for training, evaluating, and improving sarcasm detection systems.
Takeaways & Limitations
Self-annotation remains noisy because some sarcastic comments lack “/s” labels and some “/s” uses do not indicate sarcasm.
Abstract
from arXiv · showhide
We introduce the Self-Annotated Reddit Corpus (SARC), a large corpus for sarcasm research and for training and evaluating systems for sarcasm detection. The corpus has 1.3 million sarcastic statements -- 10 times more than any previous dataset -- and many times more instances of non-sarcastic statements, allowing for learning in both balanced and unbalanced label regimes. Each statement is furthermore self-annotated -- sarcasm is labeled by the author, not an independent annotator -- and provided with user, topic, and conversation context. We evaluate the corpus for accuracy, construct benchmarks for sarcasm detection, and evaluate baseline methods.
1 Introduction
Sarcasm detection matters for NLP systems but remains difficult because sarcasm is infrequent and hard for humans to identify. SARC addresses gaps in existing corpora with a large, self-annotated, context-rich Reddit dataset.
- Sarcasm detection supports natural language understanding, dialogue systems, and text mining.
- Existing datasets commonly use balanced labels or human annotations, limiting their alignment with naturally occurring sarcasm.
- More than 1 million sarcastic statements make SARC an order of magnitude larger than previous sarcasm corpora.
- Each statement includes author, topic, and conversation context while using self-annotation rather than independent labeling.
2 Related Work
Prior sarcasm corpora vary by source, annotation strategy, and label balance, motivating common evaluation baselines. SARC builds on Reddit self-annotation while avoiding some limitations of Twitter and earlier Reddit datasets.
- Prior studies using separate datasets illustrate the need for common evaluation baselines.
- Sarcasm datasets differ mainly in their data sources, amount of human annotation, and balance between sarcastic and non-sarcastic statements.
- Earlier Reddit work allowed unbalanced labels but used self-annotation less directly and produced around 10,000 human-labeled sentences.
- Twitter offers hashtag-based self-annotation and large raw data, but abbreviated language makes it a less attractive source for annotated comments.
3 Corpus Details
SARC uses Reddit’s tree-structured conversations and “/s” markers to assemble a large self-annotated corpus. The released data supports large-scale analysis and benchmark construction, while annotation noise remains a concern.
- Reddit organizes submissions and comments into tree-like conversations, with each comment linked to a parent.
- Users commonly mark sarcasm with “/s”, but the marker is noisy because some users omit it, misunderstand it, or use it when sarcasm is already obvious.
- The corpus covers comments from January 2009 through April 2017 and records labels, authors, subreddits, scores, dates, and dataset identifiers.
- 533 million comments include around 1.3 million sarcastic comments, with each row also containing the parent comment.
- The released files provide conversation threads, sibling comments, and labeled responses for constructing benchmark datapoints.
- The raw corpus supports large-scale machine learning, statistical analysis, and smaller benchmark tasks, including balanced and unbalanced regimes.
4 Corpus Evaluation
The corpus evaluation examines data quality, source characteristics, and limitations of self-annotated Reddit sarcasm. It finds that filtering reduces annotation noise but cannot eliminate contextual false negatives and other labeling errors.
- Data quality: 500 sarcastic and 500 non-sarcastic comments were manually checked to estimate false positives and false negatives from Reddit self-annotation.The evaluation used full comment context when judging labels.
- Corpus characteristics: Well-moderated and special-interest subreddits have less sarcasm than controversial and less-moderated subreddits.Figure 2 covers subreddits with more than a million comments in SARC.
- Comparison with other sources: Reddit offers longer, less hashtag-dependent text, accessible conversation context, and explicit subreddit topics compared with Twitter.The authors report .002% of Twitter authors versus .927% of Reddit authors using sarcasm annotation in a July 2014 subsample.
- Limitations: False positives arise when “/s” is used for non-sarcastic meanings, including discussing the notation or representing HTML strike-through tags.These cases can occur when users are unaware of the convention or when subreddit topics support alternate senses.
- Limitations: The authors filter users, comment placement, and sarcastic-comment descendants, but acknowledge that contextual judgment can remain unavailable to human evaluators.They use prior “/s” usage, retain terminal tags, discard child comments of sarcastic comments, and use voting among annotators.
- Limitations: False negatives are harder to detect because sarcastic comments may omit “/s” when users consider sarcasm obvious, making community and conversation context important.Sarcastic comment chains can also produce subsequent sarcastic comments without the annotation.
5 Benchmarks for Sarcasm Detection
The paper turns SARC into benchmarks for classifying sarcastic versus non-sarcastic statements. These benchmarks expose conversation context and comment metadata to learning systems while providing simple reproducible baselines.
- Benchmark construction: The benchmarks classify statements as sarcastic or non-sarcastic using full conversation threads and comment metadata.The authors construct several benchmarks from the raw corpus for training and evaluating sarcasm detection systems.
- Baseline methods: Context-free baselines use linear classification over simple featurizations, and code is provided to reproduce the results.The benchmark design follows the stated task specification.
- Benchmark construction: The benchmarks support evaluation of sarcasm detection systems across balanced and unbalanced label regimes.The supplied passage states that several useful benchmarks are constructed, while the paper context describes both regimes.
5.1 Evaluation Task
The evaluation tasks ask systems to identify sarcasm among responses within conversation context, with both general and balanced formulations. A politics-only setting tests performance on a familiar topic.
- General task: Given a post and comment sequence, systems identify which responses to the final comment are sarcastic.Each datapoint contains a conversation thread, candidate responses, and sarcasm labels.
- General task: Performance on the general task is measured with average precision, recall, and F1 scores.
- Balanced task: The balanced task pairs one sarcastic and one non-sarcastic response sharing a context, requiring systems to choose the sarcastic statement by accuracy.This construction uses responses drawn from the same set of replies to a comment sequence.
- Topic-specific task: Human and machine performance is also tested on politics comments, where evaluators have sufficient background knowledge.The setting isolates a topic familiar to all evaluators.
5.2 Methods
The evaluation uses simple lexical, embedding, random, and human baselines to distinguish sarcastic from non-sarcastic responses, with context supplied for human judgments.
- An 80%-20% train-test split evaluates three baseline approaches on two datasets.
- Bag-of-Words and Bag-of-Bigrams use document n-gram counts as classification features.For the all-subreddit subsample, features must occur at least 5 times in training comments.
- Sentence embeddings represent documents by summing 1600-dimensional GloVe word embeddings.The embeddings were trained on the Amazon product corpus.
- Human performance uses five evaluators who classify 100 samples with full conversation context, followed by majority voting.
- The random baseline labels responses independently as sarcastic using the training-set average sarcasm fraction.
5.3 Results
The baseline classifiers outperform the random baseline but remain below human performance, while lexical n-gram representations outperform sentence embeddings. Human agreement is significant but imperfect, and majority voting benefits from aggregate context.
- 5.3.1 Baselines: Baseline methods perform reasonably well and substantially better than the random baseline, but none reaches human performance on either dataset.
- 5.3.1 Baselines: Bag-of-Words and Bag-of-Bigrams outperform sentence embeddings, while distributed representations remain relevant for incorporating context.
- 5.3.2 Human: Human evaluators significantly outperform the baseline methods as both majority and average classifiers.
- 5.3.2 Human: Fleiss kappa is 0.5 on the main dataset and 0.67 on the politics subsample, indicating moderate and substantial agreement respectively.
- 5.3.2 Human: Majority voting performs especially well on the full dataset, surpassing performance on the politics subsample despite lower individual human performance.
6 Conclusion
The paper introduces a freely available, self-annotated Reddit sarcasm corpus with over 1 million sarcastic sentences and benchmarks simple machine-learning methods against human performance.
- The corpus contains over 1 million sarcastic sentences, making it larger than any existing dataset.
- The raw data and evaluation subsamples are freely available for future benchmark and method development.
- The paper evaluates simple machine-learning baselines and compares them with human performance.