Source-linked AI summary
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest
Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, Yejin Choi
TL;DR
The paper asks whether AI models understand humor by testing progressively difficult cartoon-caption tasks in multimodal and description-based settings. It finds persistent gaps from human performance while releasing annotations and tools for further research and possible creative assistance.
Problem
The paper examines whether AI models can understand humor involving subtle image-caption relations and indirect allusions to human experience and culture.
Method
The authors evaluate models on matching captions to cartoons, identifying winning captions, and explaining humor, using both pixels and human-authored visual descriptions.
Results
AI models struggle across all three tasks: the best multimodal model reaches 62% versus humans’ 94% on five-way matching, while humans outperform machine explanations in the reported comparisons.
Takeaways & Limitations
The models’ partial capabilities may support creative collaboration, including feedback on caption relevance or quality and brainstorming assistance for humorists and cartoonists.
Takeaways & Limitations
The study covers only the narrow humor slice represented by The New Yorker Cartoon Caption Contest and therefore does not represent all types of humor.
Abstract
from arXiv · showhide
Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of "understanding" a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image's locations/entities, what's unusual in the scene, and an explanation of the joke.
1 Introduction
The paper introduces three progressively harder benchmarks for humor understanding, from matching captions to cartoons to judging caption quality and explaining jokes. These tasks target indirect image-caption relationships and cultural or experiential allusions, and reveal substantial gaps between AI and humans.
- Benchmarks: AI models are evaluated on matching captions to cartoons, identifying higher-quality captions, and explaining why image-caption combinations are funny.The tasks progress from recognizing relevance to evaluating quality and generating explanations.
- Challenge: Winning captions can depend on subtle visual anomalies and playful associations rather than literal image descriptions.The “Can you please pass the cow?” example requires noticing unusually large mugs and associating them with a cow’s worth of milk or cream.
- Resources: The released resources include benchmarks, annotations, models, code, a leaderboard, and a corpus describing locations, entities, unusual elements, and joke explanations.The resources are intended to support further research and human-AI collaboration tools.
- Results: 62% accuracy versus 94% for humans on the 5-way matching task shows a large multimodal performance gap.The best multimodal model was fine-tuned CLIP ViT-L/14.
- Results: Human explanations were preferred in more than two-thirds of comparisons with 5-shot GPT-4, even when models received detailed visual annotations.The comparison used the best explanation model and human-authored explanations.
2 Datasets and Task Setups
The paper builds a corpus from 14 years of New Yorker caption contests and defines matching, quality-ranking, and explanation tasks. It evaluates models either from cartoon pixels or from rich human-written descriptions and annotations.
- Dataset: The corpus compiles 14 years of weekly contests containing cartoons, entries, three editorial finalists, and crowd quality estimates for some contests.The sources include roughly 250 contests with about 1.5 million captions and an earlier set covering contests #1–#507.
- Dataset: The benchmark extends prior work with matching, quality ranking, explanation tasks, dense cartoon annotations, and 651 author-written joke explanations.Table 1 identifies these as the dataset and task contributions.
- Matching: Matching selects the one caption appropriate for a cartoon from five choices, with distractors drawn from finalists of other contests.The distractors are balanced so answer choices alone cannot exceed chance accuracy.
- Quality ranking: Quality ranking asks models to distinguish an official finalist from a non-finalist caption written for the same cartoon.The evaluation reports NYAcc for official finalists and CrowdAcc for captions rated highly by crowds.
- Explanation: Explanation evaluates free-form accounts of why a high-quality caption is funny against 651 human-created explanations.The guidance asks authors to explain the joke to a friend who does not understand it yet.
- Experimental settings: From Pixels models process only the cartoon image, whereas From Description models receive human annotations that simulate access to human-level visual understanding.Annotations cover settings, literal scenes, unusual elements, and relevant Wikipedia-linked entities.
3 Experiments
The experiments evaluate vision-language and language models on New Yorker caption-contest tasks using held-out contests and two input settings: images alone or human-authored descriptions.
- Experimental setup: Models are evaluated across five cross-validation splits that hold entire contests out at test time.The experiments include CLIP, OFA→language-model, T5, GPT-3/3.5/4, and caption-only baselines.
- Models: CLIP uses image-text similarity for multiple-choice prediction but cannot generate explanations because it is not generative.The fine-tuned CLIP model is a 428M-parameter vision-language Transformer, while OFA Huge maps cartoons and prompts to structured descriptions for a language model.
- Experimental setup: From Pixels models process only cartoon images at test time, whereas From Description models receive human-written annotations simulating human-level visual processing.Multiple-choice tasks concatenate descriptions and choices; explanation models generate text conditioned on descriptions and captions.
- Evaluation: Human performance estimates come from three people attempting 100 randomly sampled matching and quality-ranking instances with access to images but not descriptions.The authors caution that human performance is not necessarily an upper bound for preference labels because tastes vary and labels aggregate editor or crowd judgments.
CLIP✘
Across matching, quality ranking, and explanation evaluations, models often benefit from visual or descriptive context but remain uneven against human preferences and image understanding.
- 3.1 Matching and quality ranking results: GPT-4 generally performs best among From Description models, reaching 84.5% accuracy on matching, but its CrowdAcc of 73.3 trails humans’ 83.7.GPT-4 and fine-tuned GPT-3 exceed the human estimate for New Yorker editor selections, while crowd-selection prediction remains weaker.
- 3.1 Matching and quality ranking results: Models mostly outperform the Caption Only baseline, suggesting that cartoon-caption feature interactions improve prediction beyond caption content alone.Fine-tuned CLIP is strongest for From Pixels matching, while OFA+T5-11B is competitive for quality ranking and supports generation.
- 3.2 Human evaluation of explanation.: 84.7% of pairwise cases favored T5-11B with image information over the same model without image information.This comparison directly tests whether image context improves explanation generation.
- 3.2 Human evaluation of explanation.: In explanation generation, larger T5-11B is preferred over T5-Large in 68.5% of cases, while fine-tuning and in-context learning are comparable.GPT-4 benefits from supervision: 5-shot GPT-4 is preferred to zero-shot GPT-4 in 64% of cases, and to 5-shot GPT-3 in 93%.
- 3.2 Human evaluation of explanation.: Human-written explanations were preferred in 68% of pairwise cases over few-shot GPT-4 explanations.The authors report recurring image-interpretation errors in machine explanations, while unanimous GPT-4 wins were rare.
4 Related Work
Related work spans humor theory, New Yorker caption-contest modeling, multimodal humor recognition, caption preference measurement, and explanations of humor.
- Humor.: Humor theories commonly emphasize hostility, release of constraints, or incongruity and its resolution.These traditions frame humor through superiority, relief, or incompatible contexts that are subsequently resolved.
- NLP + The Caption Contest: Prior New Yorker caption-contest studies model funniest-caption selection using features such as image-setting match, uncanniness, readability, lexical relations, and sentiment.The present corpus extends related annotations and modeling resources while adding tasks and audience-preference distinctions.
- Measuring preferences over captions: Caption-preference research has modeled average rater preferences, including quality ranking and bandit-style selection, but some systems do not directly use cartoon content.The paper evaluates content-aware prediction and distinguishes New Yorker editor preferences from crowd preferences.
- Multimodal and computational humor: Multimodal humor research covers humor recognition in images, laughter prediction in spoken or scripted media, political cartoons, and figurative-language inference.These lines of work provide neighboring approaches rather than the paper’s specific three-task benchmark.
- Explaining humor: Humor-explanation work treats joke explanations as accounts of the proximal mechanism behind a predicted label, with prior qualitative study focused on non-visual jokes.The paper applies explanation evaluation to image-caption combinations.
5 Conclusion
The study finds that current vision and language models remain less effective than humans at recognizing, evaluating, and explaining humor in The New Yorker Caption Contest, while retaining potential as creative collaborators.
- The study releases benchmarks, annotations, models, a leaderboard, and code to support further research and human-AI collaboration tools.
- Current vision and language models still cannot recognize caption relevance, reproduce crowdsourced humor rankings, or explain contest humor as effectively as humans.
- AI’s substantial partial capacity may support creative collaboration, including brainstorming assistance and quantitative feedback for humorists and cartoonists.
- The authors cannot determine when, or whether, the gap between human and machine humor understanding will close.
6 Limitations
The study’s conclusions are bounded by the narrow cultural scope of the New Yorker contest, the interpretation of editorial rankings, and limited diversity in explanation authorship.
- The contest represents a narrow slice of humor tied to one language, region, history, culture, style, and set of conventions.
- Editorial selections are not objective ground truth for funniness, and the ranking task models average preferences that may reflect prior contest conventions.
- Because most corpus explanations were written by one author, future work should examine person-to-person variation in explaining humor.
- The annotation studies used qualified Mechanical Turk workers and separate interfaces for descriptions, links, and pairwise explanation judgments.
- The released annotations hash crowdworker IDs so they cannot be traced back to individual workers.
B Additional Experimental Details
Additional experiments use controlled annotation sampling, image and language-model training procedures, and prompt-based evaluations across the study’s experimental settings.
- At test time, each model receives one randomly sampled annotation set, held constant across methods for fair comparison.
- The annotation design permits 36 location, description, uncanny-description, and entity-link combinations while preserving annotator-level dependencies.
- Fine-tuning used validation early stopping, cross-validation-specific learning-rate choices, image augmentation, and model-specific training settings.
- Cartoon-description HITs collected locations, ordinary descriptions, unusual-scene descriptions, and entities through four explicit questions.
- Directly fine-tuning OFA for explanation generation produced roughly 300 perplexity and errors that were difficult to diagnose.
- GPT experiments used zero-shot and five-example in-context prompts, while GPT-4 queries cost approximately $4K in total.
C Task Construction Details
The paper constructs three humor-understanding tasks from New Yorker caption contests and supports them with curated caption sets, matching instances, and prompt-based model evaluations.
- Caption selection: 2.7K high quality captions across 704 contests anchor the benchmark, with each contest contributing 3 or 6 selected candidates.Official finalists are supplemented by highly crowd-rated captions, followed by SBERT-based semantic deduplication.
- Task prompts: The benchmark includes matching, quality ranking, and explanation prompts that ask models to relate jokes to uncanny scenes and explain their humor.The matching prompt presents five choices with one scene-corresponding caption; explanation prompts require both relation and funniness.
- Model prompting: GPT-3 experiments compare zero-shot, five-shot, and fine-tuned settings, while GPT-4 uses structured chat prompts with system instructions and option-by-option reasoning.The GPT-3 prompt examples cover matching and quality ranking, and the scaling experiment reports text-davinci-002 as the strongest GPT-3 model.
- Instance construction: Matching instances pair each high quality caption with one correct cartoon and four mismatched options while preventing cross-split caption overlap and answer-only biases.Negative assignment occurs within cross-validation splits, and each answer appears once correctly and four times incorrectly.
- Instance construction: Quality-ranking instances sample “okay” captions by removing duplicates and using crowd-rating tertiles or a caption-only quality model when ratings are unavailable.Earlier contests use a model trained on crowd-labeled data to filter clearly low-quality captions before sampling.
- Explanation examples: Caption explanations depend on indirect visual associations and cultural or linguistic allusions, such as interpreting “first monkey” and “third monkey” as narrative perspectives.The examples illustrate why literal scene recognition alone does not fully capture the joke.
D Graphical version of matching and ranking results.
Figures 12 and 13 visualize matching and quality-ranking results across zero-shot, five-shot, and fine-tuned models, with human results marked separately and a caveat about crowd-based comparisons.
- Matching results: Figure 12 compares matching results using vertically stacked bars for zero-shot, five-shot, and fine-tuned models, with human results shown as dark green lines.The visual distinguishes model prompting or training conditions while retaining a human reference.
- Ranking results: Figure 13 displays quality-ranking results in a scatter plot using the same model-condition conventions as Figure 12.The plot compares model results against the corresponding human-based reference axes.
- Evaluation caveat: Crowd accuracy may be statistically more reliable than New Yorker-editor accuracy because many more crowd selectors contribute to the former.The caveat concerns the relative sample sizes underlying the two evaluation preferences.
E Automatic evaluation of explanations
The paper reports automatic explanation-evaluation metrics in Table 5, while judging human evaluations to be better quality measures.
- Table 5 reports automatically calculated metrics for evaluating explanations.
- Perplexity is unavailable for GPT-3.5 and GPT-4 because their log probabilities are unavailable.
- The authors consider the human evaluations in the main text better quality measures than the automatic metrics.
F Machine explanations that were preferred over human ones
A minority of machine explanations were preferred over human references, usually when they were more succinct or supplied a more meaningful detail while remaining topical.
- GPT-4 comparisons: In 8/130 GPT-4 five-shot cases, all three annotators preferred the machine explanation to the human reference.The authors’ close reading found both explanations topical, with machine advantages often involving succinctness or meaningful detail.
- Interpretation: The preferred machine explanations were generally judged reasonable alongside the human explanations rather than categorically superior.The reported GPT-3 and GPT-4 cases emphasize topical adequacy, with advantages concentrated in concision or specific details.
- GPT-3 comparisons: GPT-3 explanations were preferred by a majority of annotators in 16/131 cases, including one case with unanimous machine preference.In the other 15 cases, two of three annotators preferred the machine generation.
G Aiding humor generation with system-assisted brainstorming
The paper repurposes its annotated corpus into a structured prompt for brainstorming cartoon situations, caption concepts, captions, and explanations. Demonstrations suggest language models can generate creative cartoon-caption combinations, including both unconditional and description-conditioned outputs.
- G Aiding humor generation with system-assisted brainstorming: The curated prompt reframes corpus annotations to support cartoon descriptions, concept brainstorming, caption generation, and explanations.Its stages include scene descriptions, uncanny elements, entities, caption concepts, selected ideas, finalist captions, and explanations.
- G Aiding humor generation with system-assisted brainstorming: The prompt organizes brainstorming by moving from a scene description and unusual element to entities, caption concepts, a caption, and an explanation.An example uses a kennel scene, human occupants, and pet sterilization concepts to produce a caption and explain its relation to the scene.
- G Aiding humor generation with system-assisted brainstorming: A 2,407-token prompt reformulates corpus annotations for creative collaboration on unusual situations, contest concepts, and captions.The figure presents line groups intended to support successive stages of ideation and caption writing.
- G Aiding humor generation with system-assisted brainstorming: Within five samples, GPT-3 generated an unconditional scene about a chicken playing croquet with the caption “I’m not sure this is what they meant by free range.”The authors also describe conditional generation from a real contest scene description.