Source-linked AI summary
Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue
Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, Marilyn Walker
TL;DR
Sarcasm’s diverse cues and the difficulty of operationalizing and reliably annotating it have constrained high-quality dialogue corpora. This paper builds a larger, more diverse corpus, operationalizes rhetorical-question and hyperbole subtypes, and reports improved simple-feature classification and high-precision subtype patterns.
Problem
Sarcasm’s diverse forms and cues make it difficult to operationalize computationally in dialogue, while low prevalence and annotation challenges complicate corpus construction.
Source: §1 · p. 1Method
Results
The corpus achieves a best F of 0.74 with simple features, while weakly supervised learning yields high precision but low recall for rhetorical-question and hyperbole datasets.
Source: §5 · p. 9Takeaways & Limitations
The high-precision patterns for rhetorical questions and hyperbole may be used to bootstrap these two classes.
Source: §5 · p. 9Evaluation on real-world distributions with inherently unbalanced sarcastic and not-sarcastic posts remains future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The use of irony and sarcasm in social media allows us to study them at scale for the first time. However, their diversity has made it difficult to construct a high-quality corpus of sarcasm in dialogue. Here, we describe the process of creating a large- scale, highly-diverse corpus of online debate forums dialogue, and our novel methods for operationalizing classes of sarcasm in the form of rhetorical questions and hyperbole. We show that we can use lexico-syntactic cues to reliably retrieve sarcastic utterances with high accuracy. To demonstrate the properties and quality of our corpus, we conduct supervised learning experiments with simple features, and show that we achieve both higher precision and F than previous work on sarcasm in debate forums dialogue. We apply a weakly-supervised linguistic pattern learner and qualitatively analyze the linguistic differences in each class.
1 Introduction
Sarcasm’s diverse cues and the difficulty of operationalizing it computationally and collecting reliable annotations motivate a larger, more diverse dialogue corpus. The paper introduces rhetorical-question and hyperbole subcorpora and reports improved classification results with simple features.
1 Introduction
- Incongruity features improved sarcasm classification on tweets but not dialogue, while sarcasm’s many cues make dialogue operationalization challenging.Earlier IAC work reported best precision values of 0.62, 0.66, and around 0.70, with best F values of 0.57, 0.70, and 0.64, respectively.Source: §1 · p. 1
- The authors sought a larger, more diverse corpus because sarcastic utterances are rare in debate forums and sarcasm annotation is difficult to make reliable.They filter data to increase the share sent for annotation and report high annotation reliability.Source: §1 · p. 1
- The paper operationalizes rhetorical questions and hyperbole as distinct sarcasm subtypes and uses a weakly supervised learner to characterize their linguistic patterns against generic sarcasm.The analysis describes differences across generic sarcasm, rhetorical questions, and hyperbole.Source: §1 · pp. 1–2
- 0.73 precision and 0.74 F are the highest reported on the new corpus with basic n-gram and Word2Vec features, improving on previous work.The experiments use simple features to demonstrate corpus quality.Source: §1 · p. 2
- High-precision NOT-SARCASTIC classifiers are straightforward to develop across the rhetorical-question and hyperbole subtypes in debate-forum dialogue.The paper attributes this result to the nature of these utterances in that setting.Source: §1 · p. 2
2 Creating a Diverse Sarcasm Corpus
The paper constructs a larger, more diverse sarcasm corpus from online debate and operationalizes rhetorical questions and hyperbole as distinct subcorpora. Filtering and annotation expand the generic set while reducing annotation effort, and cue-based collection creates balanced subtype datasets.
2 Creating a Diverse Sarcasm Corpus
- The corpus operationalizes rhetorical questions and hyperbole as distinct subtypes, which prior work had not concretely operationalized for comparison with generic sarcasm.The final rhetorical-question set has 851 posts per class; the hyperbole set has 582 posts per class.Source: §2 · p. 2, §2.2 · pp. 5–6
2.1 Generic Dataset (Gen)
- AutoSlog-TS extracted high-precision NOT-SARCASTIC patterns, and the authors used them to filter candidate posts before quote-response annotation.The learner classified NOT-SARCASTIC posts with 80–90% precision; the authors note that filtering might also remove sarcastic posts.Source: §2.1 · pp. 2–3
- The generic corpus contains 3,260 posts per class, or 6,520 total, after expansion with posts labeled sarcastic by at least 2 of 3 annotators.The initial addition used a stricter threshold of at least 6 of 9 sarcasm labels.Source: §2.1 · p. 3
- Filtering 30,000 posts down to about 11,000 for annotation raised the sarcastic share to 20% and reduced annotation to one-third of the original pool.The conservative threshold was at least 6 of 9 sarcasm labels; the authors say including posts with 5 of 9 labels could raise the ratio to 31%.Source: §2.1 · pp. 3–4
2.2 Rhetorical Questions and Hyperbole
- The middle-of-post question-followed-by-statement heuristic yielded 75% majority-vote rhetorical questions in a 100-post check, supporting retrieval without manually annotating every candidate.The final rhetorical-question corpus combines 357 generic-corpus posts with cue-matched posts labeled by five annotators.Source: §2.2 · pp. 4–5
3 Learning Experiments
The experiments use simple supervised features and weakly supervised linguistic patterns to examine corpus quality and differences across subcorpora. The Gen corpus reaches the paper’s best reported F-measure, while AutoSlog-TS shows higher precision is attainable for RQ and Hyp than for Gen.
Supervised Learning
- The supervised experiments use a linear SVM with SGD and L2 regularization, 10-fold cross-validation, and n-grams or Google News Word2Vec; custom dialogue embeddings did not work well for sarcasm classification.The authors frame these experiments as exploring corpus properties and quality, not optimizing classification.Source: §3 · p. 6
- 0.74 F-measure for SARCASTIC with Google News Word2Vec and 0.73 for NOT-SARCASTIC with n-grams on Gen exceed prior dialogue results.Table 7 reports 10-fold cross-validation results on balanced classes.Source: Table 7 · p. 6, §3 · p. 6
- 0.70 and 0.71 are the best SARCASTIC and NOT-SARCASTIC F-measures on RQ; Hyp reaches 0.65 and 0.68, respectively, with n-grams.For RQ, n-grams include repeated punctuation and emoticons that the Word2Vec model excludes; Hyp results are lower, likely because its dataset is smaller.Source: §3 · p. 6
Effect of Dataset Size
- As dataset size grows, performance generally improves; RQ is comparable to Gen, which reaches 0.74 sarcastic F-measure at 3,260 posts per class.The comparison holds posts per class constant; the plot includes up to 851 posts per class for RQ and 582 for Hyp.Source: Figure 3 · p. 6, §3 · p. 7
Weakly-Supervised Learning
- AutoSlog-TS uses syntactic templates, including added adjective-noun, adverb-adjective, and adjective-adjective patterns, with frequency, class-probability, and post-level thresholds tuned by grid search.The pattern templates must satisfy their associated syntactic constraints.Source: §3 · p. 7, Table 8 · p. 7
- Learned Gen patterns show contrast drawn from varied sources, such as differences between what someone was asked and what they did.Other examples contrast social groups, chapters, and what one person is allowed to do versus what another must do.Source: §3 · p. 7, Table 8 · p. 7
- At precision 0.75, AutoSlog-TS yields recall 0.07 for RQ and 0.08 for Hyp; both subcorpora can attain higher precision than Gen.The reported recall is low, though the authors note retrieved posts often contain multiple cues and web datasets are large.Source: §3 · p. 7, Figure 4 · p. 8
4 Linguistic Analysis
AutoSlog-TS and dependency parsing are used to characterize patterns that distinguish sarcastic from non-sarcastic language and to examine generic sarcasm, rhetorical questions, and hyperbole. The learned patterns reveal both class-specific cues and diverse sources of contrast.
Analysis Method
- AutoSlog-TS extracts linguistic patterns, and dependency parsing examines argument relations; the analysis counts patterns meeting frequency 2 and probability 0.75 thresholds.The authors report learning many novel lexico-syntactic cues beyond the regex patterns used for retrieval.Source: §4 · p. 8, Table 10 · p. 8
Generic Sarcasm
- Generic sarcastic patterns tend toward subjective, topic-independent language, while non-sarcastic patterns capture technical and scientific language.Adjective and adverb patterns are abundant among sarcastic patterns and can co-occur with the retrieval cues.Source: §4 · p. 8
Rhetorical Questions
- Rhetorical-question sarcasm often attacks the addressee’s mental abilities, as shown by the extracted verb, subject, and object relations.The non-sarcastic RQ patterns resemble topic-specific non-sarcastic patterns in Gen.Source: §4 · p. 8, Table 11 · p. 9
Hyperbole
- Hyperbole yields frequent, high-precision adverb-adjective cues and verbal patterns linked to contrast, including exclusion and expansion of a predicated class.These patterns were not used to retrieve hyperbole, but they appear in the learned pattern inventory.Source: §4 · p. 9, Table 12 · p. 9
5 Conclusion and Future Work
The paper presents a large, diverse sarcasm corpus and reports supervised and weakly supervised results alongside qualitative analyses of sarcasm subtypes. It identifies future work in feature engineering and evaluation on naturally unbalanced real-world data.
Corpus and Annotation
- Filtering and crowd-sourced annotation produce a large, diverse corpus, with sarcastic posts skewed to 20–31% rather than the estimated 12% forum prevalence.The annotation process uses a conservative threshold of at least 6 of 9 annotators agreeing that a post is sarcastic.Source: §5 · p. 9
- Operationalized lexico-syntactic cues are highly indicative of sarcasm, with ratios up to 87%, across generic sarcasm, rhetorical questions, and hyperbole.The final corpus represents all three categories.Source: §5 · p. 9
Learning Results
- Simple supervised features achieve a best F of 0.74; weakly supervised patterns reach higher precision for RQ and Hyp than Gen, with low recall.The high-precision RQ and Hyp patterns may be useful for bootstrapping those classes.Source: §5 · p. 9
Linguistic Analysis
- Qualitative analysis finds diverse forms of sarcasm and diverse sources of the underlying semantic contrast in dialogue.The analysis covers differences in rhetorical questions, hyperbole, and generic sarcastic versus non-sarcastic cues.Source: §5 · p. 9
Future Work
- Future work targets feature engineering and evaluation on real-world distributions where sarcastic and non-sarcastic posts are inherently unbalanced.The authors also aim to better characterize and model sarcasm’s diversity in dialogue.Source: §5 · p. 9