Source-linked AI summary
Jokes Aside: Measuring the Semantic Distance of Double Meanings
Fabio De Ponte
TL;DR
Automated humor recognition remains difficult, and existing embedding-based predictors do not clearly test theories about humor mechanisms. This paper revisits humor metrics with embeddings, introduces symmetry, and evaluates them across three datasets. Performance is generally poor, although symmetry is consistently associated with higher-rated jokes and may be necessary but not sufficient for effective humor.
Problem
Embedding-based humor models lack interpretability for testing theories about the mechanisms underlying humor.
Method
The paper calculates proposed humor metrics from OpenAI and MiniLM embedding vectors, using JokeJudger, Expunations, and rJokes, with paired interpretation sentences added to the latter two datasets.
Results
Models trained on the proposed metrics performed poorly across datasets: JokeJudger reached 57.1% accuracy versus a 61.5% baseline, while Expunations and rJokes reached 51.1% and 41.8%.
Takeaways & Limitations
Symmetry is consistently associated with higher-rated jokes, suggesting it may be necessary but not sufficient for effective humor.
Takeaways & Limitations
The study excludes the inappropriateness metric, which is identified as one factor that can heighten humor.
Abstract
from arXiv · showhide
Large language models have significantly enriched the toolkit for computational humor research, particularly in the automated generation of jokes and puns. A key innovation, contextual embedding vectors, offers new opportunities to revisit and refine earlier hypotheses. Notably, Petrovic and Matthews (2013) proposed a joke generation model based on the scheme "I like my X like I like my Y, Z" (e.g. "I like my ice like I like my dreams, crushed"). They suggested that joke hilarity increases with: a) frequent association of Z with X and Y, b) rarity of Z, c) ambiguity of Z, and d) meaning distance between X and Y. Building on this, Winters et al. (2019) proposed a set of metrics, based on Google Ngrams and Word2Vector. In this work, three out of their five metrics are revisited with word embeddings: obviousness, compatibility, and comparison. Another measure, symmetry, defined as closeness of Z to both X and Y, is introduced here for the first time. Two models were used to collect the embedding vectors (OpenAI text-embedding-3-small and MiniLM all-MiniLM-L6-v2) on three datasets: JokeJudger, Expunations, and rJokes. The last two datasets, Expunations, and rJokes, were expanded by adding paired sentences that captured the ambiguous expression at the core of each joke in its two different meanings. Results revealed that models trained on the proposed metrics performed poorly in predicting humor ratings: on JokeJudger, the best model achieved 57.1% accuracy, below the 61.5% baseline, while performance on Expunations and rJokes was even lower. Nevertheless, the symmetry metric seems consistently associated with higher-rated jokes, suggesting it may capture a necessary -though not sufficient- property of humor.
INTRODUCTION
Humor is widespread in human communication and online life, yet automated humor recognition remains difficult because humor is subjective and people often disagree about what is funny.
- Humor permeates online communication, from social exchanges and political satire to memes, irony, and viral jokes.
- Automated humor recognition remains challenging despite recent advances in natural language processing and large language models.
- Humor is subjective and constantly evolving, making it difficult to obtain reliable data for computational modeling.
BACKGROUND
Computational humor research has drawn on incongruity theories, corpus statistics, lexical resources, and word embeddings to model how jokes create and resolve double meanings. Earlier work proposed measurable properties of joke hilarity, while later studies used embeddings for semantic and contextual analysis.
- Humor theories: Incongruity theory explains humor as an apparent contradiction resolved by a punchline that reinterprets the initial conflict.
- Joke-generation models: Petrovic and Matthews proposed the “I like my X like I like my Y, Z” template and linked hilarity to association, rarity, ambiguity, and distance.
- Prior metrics: Winters et al. introduced five metrics—obviousness, conflict, compatibility, comparison, and inappropriateness—using Google Ngrams, WordNet, and DEViaNT.
- Prior metrics: Their metrics operationalized humor-related properties through word frequency, word combinations, number of meanings, adjective-vector similarity, and word sexiness.
- Embedding-based research: Word embeddings have been used to measure semantic disconnection, incorporate contextual information, represent humor preferences, and detect forms such as sarcasm and punchlines.
- Research gap: The present work addresses the interpretability gap by using embedding vectors to calculate specific metrics rather than only feeding them into rating-prediction models.
METHODOLOGY
The study operationalizes four embedding-based metrics for double-meaning jokes and evaluates them across JokeJudger, Expunations, and rJokes. Expunations and rJokes were expanded with paired sentences representing each expression’s two meanings before models were trained to predict humor ratings.
- Metric design: Four metrics—obviousness, compatibility, comparison, and symmetry—were defined for the template “I like my X like I like my Y, Z.”Obviousness and compatibility measure cosine similarity between related embeddings; comparison measures interpretive distance, while symmetry is the difference between obviousness and compatibility.
- Metric design: Symmetry measures whether the two meanings work equally well by taking the difference between obviousness and compatibility.Lower symmetry values are intended to indicate more balanced interpretations.
- Model training and evaluation: Ratings were converted into five star classes, embeddings were calculated for the relevant texts, and random forest, SVM, naive Bayes, and regression models were trained.For JokeJudger, embeddings were calculated for X, Y, Z, XZ, and YZ; the highest reported test accuracy was 57.1% from a 5-fold-cross-validated random forest.
- Datasets and representations: Expunations and rJokes were expanded with each joke’s central expression and two serious sentences representing its two implied meanings.The process produced 1,895 expanded Expunations puns and 479 expanded rJokes.
- Model training and evaluation: On Expunations, the best reported model was an SVM with 51.1% test accuracy.The results are presented in Table 2.
DISCUSSION
Embedding-based humor metrics did not predict ratings reliably across datasets, although symmetry remained concentrated near zero among higher-rated jokes. Several dataset, representation, and scope limitations may explain this weak performance.
- 57.1% accuracy was achieved by the best JokeJudger model, below Winters et al.’s 61.5% baseline.
- The embedding-based metrics showed no clear correlation with JokeJudger humor ratings, while regression produced R² = -0.02.
- 51.1% accuracy on Expunations and 41.8% on rJokes were below baseline, and both datasets yielded negative regression R² values.
- Poor prediction may reflect heterogeneous joke types, including wordplay, perspective shifts, stereotypes, celebrity references, and sexual innuendo.
- Heterographic homophones can lose written ambiguity, causing embeddings to represent only one meaning and distort metrics requiring balanced interpretations.
- Interpretation is also constrained by potentially incomparable contextual embeddings, model suitability, dataset noise, and ratings that may not represent the broader population.
- Across all three datasets, higher-rated jokes tended to have symmetry near zero, where obviousness and compatibility were nearly equal.
- Symmetry may be necessary but is not sufficient for effective humor, since lower-rated jokes showed a wider range of symmetry values.
CONCLUSION
The paper operationalizes humor-theory concepts with four embedding-based metrics and evaluates them across three joke datasets. These metrics perform poorly overall, although near-zero symmetry consistently characterizes highly rated jokes and may be necessary but insufficient for humor.
- Metrics: The four proposed metrics are obviousness, compatibility, comparison, and symmetry, replacing or excluding two metrics from Winters et al. (2019).Conflict was replaced by symmetry, while inappropriateness was discarded; word-embedding vectors replaced Google Ngrams and Word2Vec resources.
- Metrics: Obviousness measures intuitive initial interpretation, compatibility measures alignment with a hidden meaning, and comparison measures whether two distinct readings exist.
- Metrics: Symmetry measures how evenly both interpretations are expressed, especially when the same word or structure supports both.It is defined as the difference between obviousness and compatibility, with lower values expected to support effective puns.
- Evaluation: The study expanded selected Expunations and rJokes items with two non-humorous sentences representing the core double meaning, then trained four predictive models.The models were Random Forest, SVM, Naive Bayes, and Linear Regression, using the four calculated metrics to predict humor ratings.
- Results: 57.1% accuracy was achieved by the best JokeJudger model versus a 61.5% baseline, while Expunations and rJokes reached only 51.1% and 41.8%.Regression analyses found no correlation between the metrics and humor ratings.
- Results: Across datasets, the highest-rated jokes clustered near zero symmetry, suggesting symmetry may be necessary but not sufficient for highly rated humor.The authors attribute poor prediction partly to heterogeneous joke types that the metrics do not uniformly capture.
- Implications: More advanced neural approaches and large language models may capture humor mechanisms, but their lack of interpretability limits explanatory insight into how humor works.The paper therefore retains value in developing simpler, theory-oriented techniques despite the efficiency of language models.
FURTHER WORK
The paper proposes cleaner, more controlled datasets, additional metrics, human-authored meaning explanations, and direct comparisons with earlier simpler methods. These extensions target dataset heterogeneity, omitted humor features, and possible weaknesses in machine-generated representations.
- Dataset refinement: Filtering Expunations to retain mainly wordplay-based jokes could provide a cleaner test of the four metrics.The proposed filtering excludes heterographic homophones, stereotypes, celebrity references, and sexual innuendo.
- Metric extensions: Reintroducing inappropriateness would test whether taboo or norm-violating content correlates with perceived humor and rating predictions.
- Data annotation: Replacing machine-generated explanatory sentences with human-authored sentences would assess whether human interpretations improve metric reliability.
- Comparative evaluation: Earlier Google Ngram- and WordNet-based techniques outperformed the embedding methods, motivating a parallel evaluation on the same jokes.Such a comparison could clarify why the simpler metrics achieved better predictive performance.