Source-linked AI summary
Are Word Embedding-based Features Useful for Sarcasm Detection?
Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, Mark Carman
TL;DR
Existing sarcasm-detection approaches can miss subtle context incongruity, especially when sentiment words are absent. The paper augments four prior feature sets with word-embedding similarity and discordance features across four embedding types, finding broadly improved detection and stronger benefits from Word2Vec and dependency-based embeddings.
Problem
Existing sarcasm-detection approaches may miss subtle context incongruity, particularly in sarcastic text without sentiment-bearing words.
Method
The paper augments four prior sarcasm-detection feature sets with similarity- and discordance-based features from four types of word embeddings.
Results
Word-embedding augmentation improves F-score by around 4% for three of four feature sets with Word2Vec, with minor degradation for the fourth; Word2Vec and dependency-based features outperform LSA and GloVe.
Takeaways & Limitations
Word-embedding features provide a broadly useful complement to prior sarcasm-detection feature sets, with Word2Vec and dependency-based embeddings showing the greatest benefit.
Takeaways & Limitations
Contextual sarcasm and metaphors in non-sarcastic text remain challenging cases for the similarity-based approach.
Abstract
from arXiv · showhide
This paper makes a simple increment to state-of-the-art in sarcasm detection research. Existing approaches are unable to capture subtle forms of context incongruity which lies at the heart of sarcasm. We explore if prior work can be enhanced using semantic similarity/discordance between word embeddings. We augment word embedding-based features to four feature sets reported in the past. We also experiment with four types of word embeddings. We observe an improvement in sarcasm detection, irrespective of the word embedding used or the original feature set to which our features are augmented. For example, this augmentation results in an improvement in F-score of around 4\% for three out of these four feature sets, and a minor degradation in case of the fourth, when Word2Vec embeddings are used. Finally, a comparison of the four embeddings shows that Word2Vec and dependency weight-based features outperform LSA and GloVe, in terms of their benefit to sarcasm detection.
1 Introduction
Sarcasm detection depends on capturing context incongruity, but sentiment-based pattern methods can miss subtle sarcasm without sentiment words. The paper proposes augmenting prior features with word-embedding similarity and discordance.
- Context incongruity is central to sarcasm, which expresses contempt or ridicule.
- Sentiment-based pattern extraction may miss subtle sarcastic remarks that contain no sentiment-bearing words.The paper illustrates this limitation with examples involving a listener’s sense of humor and unrelated concepts.
- Word embeddings provide similarity and discordance features intended to capture context incongruity when sentiment words are absent.The approach treats semantic dissimilarity between word groups, such as ‘sense of humor’ and ‘garbage man,’ as a sarcasm clue.
- The study tests whether augmenting four prior feature sets with word-embedding features improves sarcasm detection across four embedding types.
3 Background: Features from prior work
The paper augments four previously reported sarcasm-detection feature sets, spanning lexical, dictionary-based, stylistic, and incongruity-oriented features.
- The Liebrecht et al. feature set uses unigrams, bigrams, and trigrams.
- The Gonz´alez-Ib´anez et al. feature set combines unigrams with LIWC emotion and psychological-process words.
- The Buschmeier et al. feature set adds hyperbole, punctuation patterns, interjections, and laughter expressions to unigrams.
- The Joshi et al. feature set combines implicit incongruity patterns with explicit incongruity features such as sentiment flips and polarity.
4 Word Embedding-based Features
The proposed features derive sentence-level signals from cosine similarities between content-word embeddings. They summarize the most similar and dissimilar word pairs, optionally weighting similarity by word distance.
- The proposed word-embedding features are designed to be augmented to feature sets from prior sarcasm-detection work.
- The method computes cosine similarity between embedding vectors for word pairs in a sentence.The paper illustrates the computation using all content-word pairs in the sentence ‘A woman needs a man like a fish needs a bicycle’.
- Unweighted similarity features return four values based on the maximum and minimum scores among the most similar and most dissimilar word pairs.
- For the example sentence, the maximum score of the most similar word pair is 0.766, derived from the man-woman pair.
- Distance-weighted features divide each pairwise similarity score by the square of the words’ sentence distance.This gives greater weight to similarities between terms that are close together.
5 Experiment Setup
The experiment uses a GoodReads quote dataset labeled from user tags and evaluates prior features alongside their augmented versions with five-fold cross-validation.
- The dataset contains GoodReads quotes tagged ‘sarcastic’ or ‘philosophy’ as sarcastic and non-sarcastic examples, respectively.Labels are based on user-provided tags.
- The study reports five-fold cross-validation results on the constructed dataset.
- For each of four prior papers, the experiment compares the original feature set with augmented configurations.
1. Features given in paper X
The experiments augment prior feature sets with unweighted, weighted, or combined word-embedding similarity features, alongside four embedding types.
- Four augmentation variants add unweighted similarity, weighted similarity, or both to the feature set from paper X.
- The combined variant uses both unweighted and weighted similarity features.
- The study compares LSA, GloVe, dependency-weighted vectors, and Word2Vec embeddings.
3. Dependency Weights: We use pre-trained vectors7
The embedding configurations include dependency-weighted vectors and Word2Vec vectors, with software libraries used to access the pretrained representations and compute features.
- Dependency-weighted vectors are learned using dependency distance, following Levy and Goldberg (2014).
- Word2Vec uses pretrained Google News vectors with a vocabulary of 3,000,000 words.
- Scikit supports interaction with the first three pretrained vector sets, while gensim supports Word2Vec vectors and feature computation.
6 Results
Word-embedding similarity features are weak on their own but generally improve four prior sarcasm-detection feature sets across embedding types, with gains varying by embedding and feature configuration.
- 72.53% F-score from unigrams exceeds 69.49% for unweighted similarity and 58.26% for weighted similarity used alone.
- Word-embedding features alone are therefore not sufficient and are evaluated as augmentations to four prior feature sets.
- 4% F-score improvement occurs for Liebrecht et al. (2013) with Word2Vec, while González-Ibáñez et al. (2011a) shows a 0.91% degradation.
- 76.61% to 78.09% F-score improvement occurs for Buschmeier et al. (2014) with Word2Vec.
- Maximum improvements for Liebrecht et al. (2013) are around 4% with LSA, 5% with GloVe, 6% with dependency-weighted embeddings, and 4% with Word2Vec.
- On the common-vocabulary intersection, dependency-based embeddings give a 1.048% average F-score improvement and Word2Vec gives 1.143%.
7 Error Analysis
The error analysis identifies three failure categories: incorrect word senses, sarcasm requiring conversational context, and figurative language in non-sarcastic text.
- Error categories: Incorrect word senses can cause embeddings to misclassify sarcastic statements.The paper illustrates this with “Relationship advice from one of America’s most wanted.”
- Error categories: Contextual sarcasm may evade the system when interpretation depends on the complete conversation.For “Oh, and I suppose the apple ate the cheese,” the similarity scores do not capture the conversational basis of sarcasm.
- Error categories: The system can incorrectly label non-sarcastic metaphors as sarcastic because figurative comparisons may have low semantic similarity.The paper gives a romantic metaphor involving a ripple and an ocean as an example.
8 Related Work
Sarcasm detection research has progressed from speech and lexical features toward broader feature sets, including methods that extract sarcasm-relevant patterns.
- Early approaches: Early sarcasm detection research focused on speech and lexical features.The paper cites Tepperman et al. (2006) and Kreuz and Caucci (2007) as examples.
- Pattern extraction: Later work proposed diverse additional features and increasingly emphasized extracting patterns relevant to sarcasm detection.The cited approaches include semi-supervised pattern-extraction methods.
9 Conclusion
The paper augments prior sarcasm-detection feature sets with word-embedding similarity features across four embedding types. The augmentation generally improves F-score, while Word2Vec and dependency weight-based features provide the strongest benefits relative to LSA and GloVe.
- Method: Word-embedding features use similarity scores in unweighted and distance-weighted forms, combined with four prior sarcasm-detection feature sets.The experiments use LSA, GloVe, Dependency-based, and Word2Vec embeddings.
- Results: Up to 5% F-score improvement is observed for three augmented feature sets with Word2Vec, with similar improvements for other embeddings.The paper reports improvement for three prior feature sets and notes a comparison across embedding types.
- Results: Word2Vec and dependency weight-based features outperform LSA and GloVe in their benefit to sarcasm detection.This is the paper’s cross-embedding comparison.
- Future directions: The authors suggest computing similarity over selected words or using syntactic rather than linear distance for weighting.These are proposed directions for improving the word-embedding features.