PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest

Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, Yejin Choi

arXiv:2209.06293v2cs.CLcs.CV

TL;DR

Evaluating humor understanding is difficult because cartoon captions rely on subtle, indirect connections and cultural allusions. This paper tests models on three caption-contest tasks and finds they underperform humans across all three, including on matching captions to cartoons.

  • Problem

    Caption-image links can be indirect and playful, while matching alone does not distinguish winning captions or explain why they are funny.

    Source: §1 · pp. 1–2

  • Method

    The authors test three progressively harder tasks—matching captions to cartoons, identifying winners, and explaining humor—in pixel-input and human-description settings.

    Source: §1 · p. 1, §2.1 · p. 4

  • Results

    Across all three tasks, vision-and-language models do not perform as effectively as humans; on matching, fine-tuned CLIP ViT-L/14 achieves 62% accuracy versus humans’ 94%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors suggest that despite their limitations, these models may serve as creative collaborators, including brainstorming assistants for humorists and cartoonists.

    Source: §5 · p. 9

    The ranking task predicts average rather than individual preferences, and contest selections are not ground truth for funniness and may favor traits of prior winners.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of "understanding" a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image's locations/entities, what's unusual in the scene, and an explanation of the joke.

1 Introduction

The paper introduces three increasingly demanding New Yorker cartoon-caption tasks to test AI humor understanding, finding that models lag behind humans even with rich human-authored scene descriptions.

1 Introduction

  • The benchmark tests cartoon-caption matching, distinguishing winning captions from non-finalists for the same cartoon, and explaining why a caption is funny.Matching uses finalist captions from other contests as distractors; explanation requires a free-text account of how a high-quality caption relates to the cartoon.Source: §1 · p. 1, Figure 2 · p. 2
  • The tasks are challenging because winning captions can relate indirectly to unusual visual details through playful allusions, and matching alone cannot distinguish a finalist from a funnier caption.The example “Can you please pass the cow?” depends on recognizing unusually large mugs and associating them with a large amount of milk or cream.Source: §1 · pp. 1–2
  • The benchmark evaluates models from pixels or from human-authored descriptions that cover the scene, locations and entities, unusual elements, and joke explanations.The description setting simulates access to human-level visual understanding and supports evaluating language models without built-in image processing.Source: §1 · p. 2
  • 62% accuracy by fine-tuned CLIP ViT-L/14 versus 94% for humans on the same 5-way task, while human explanations are preferred in more than two-thirds of cases over 5-shot GPT-4.The accuracy comparison is from the from-pixels setting; the explanation comparison uses the from-description setting.Source: §1 · p. 2
  • The authors release the benchmarks, annotations, models, leaderboard, and code.They also hope the work will spur progress in human-AI collaboration tools for cartoonists and contest entrants.Source: §1 · p. 2

2 Datasets and Task Setups

The section builds a two-source corpus spanning 14 years of New Yorker caption contests and defines three tasks for evaluating caption understanding, alongside cartoon annotations for image- and description-based model inputs.

2 Datasets and Task Setups

  • The corpus combines 14 years of contests: roughly 250 from Jain et al. (2020) with 1.5M captions, and contests #1–507 from Shahaf et al. (2015); Radev et al. (2016) with 2M captions.Contests include a captionless cartoon, weekly entries, three editor-selected finalists, and crowdsourced quality estimates for some contests; 55 low-resolution contests were removed from the second source.Source: §2 · pp. 2–3

2.1 Task Setups

  • Matching tests whether a model selects the one corresponding caption from five choices; distractors are finalists from other contests, and some matches require indirect or world knowledge.The task is novel, and distractors are balanced so answer-choice inspection alone cannot exceed chance accuracy.Source: §2.1 · p. 3
  • Quality ranking asks models to choose the higher-rated caption between a finalist and a non-finalist after low-quality filtering and semantic deduplication.Matching and ranking use accuracy; ranking reports NYAcc for official finalists and CrowdAcc for captions selected as high quality by the crowd, while explanation uses pairwise human evaluation.Source: §2.1 · pp. 3–4
  • The explanation task compares model-written explanations with 651 author-written explanations, using unlabeled pairs to test whether human judges prefer the author’s version.The explanations were written for randomly selected finalists after cases the author did not understand were filtered out.Source: §2.1 · p. 3

2.2 Annotation of cartoons.

  • For 704 cartoons, annotations cover scene setting, literal scene descriptions, unusualness, and relevant Wikipedia-linked entities; they serve as From Description inputs or training-only information for From Pixels models.From Pixels models receive the image at test time, while From Description models receive human-written annotations to factor out visual processing.Source: §2.1 · p. 4, §2.2 · p. 4

3 Experiments

The experiments compare image-based and description-based systems on matching, caption ranking, and explanation, using contest-held-out cross-validation. Models achieve strong results on some tasks but still fall short of human explanation preferences, and matching errors cluster by contest.

3 Experiments

  • The 704 cartoons were split into five cross-validation folds with entire contests held out for testing; model training used eight A100 GPUs.Human estimates came from three annotators who each attempted 100 randomly sampled matching and quality-ranking instances, with images available but not descriptions.Source: §3 · p. 4, Hardware+software details · p. 5, Baselines · p. 5

From Pixels (FP) Models

  • The From Pixels (FP) setup fine-tuned CLIP for matching and ranking and used OFA to predict cartoon descriptions that were passed to a language model.CLIP is not generative, so it was not used for explanation; the OFA-to-language-model composition is described as a Socratic Model.Source: From Pixels (FP) Models · pp. 4–5

From Description (FD) Models

  • From Description (FD) models used human-authored cartoon descriptions as input for text-to-text matching and ranking, and generated explanations conditioned on descriptions and captions.The study evaluated fine-tuned T5-Large and T5-11B alongside zero- and five-shot GPT-3, GPT-3.5, and GPT-4.Source: From Description (FD) Models · p. 5

3.1 Matching and quality ranking results

  • 84.5% matching accuracy was achieved by GPT-4 (5-shot); GPT-3 and GPT-4 exceeded human estimates for New Yorker selections, while GPT-4 trailed crowd accuracy.GPT-3 reached 69.8% versus the human estimate of 64.6% on New Yorker editor selections; GPT-4 reached 73.3% versus 83.7% on crowd selections. Both model settings mostly outperformed the caption-only baseline; fine-tuned CLIP led image-only matching, while OFA→T5-11B was competitive for quality ranking. Zero-shot models generally lost performance, with a smaller drop for GPT-4 chain-of-thought.Source: Table 2 · p. 6, §3.1 · p. 6

3.2 Human evaluation of explanation.

  • Image context improved explanations: T5-11B with image information won 84.7% of comparisons against the same model without it.The model using OFA-predicted descriptions lost in 74.6% of comparisons against one using human-written descriptions, and T5-11B was preferred over T5-Large in 68.5% of cases. Fine-tuned and five-shot GPT-3 explanations were comparable in pairwise judgments despite perplexities of 21.8 and 107, respectively.Source: §3.2 · pp. 6–7
  • Human-written explanations were preferred over few-shot GPT-4 explanations in 68% of pairwise cases.With equal supervision, GPT-4 explanations were preferred over GPT-3 in 93% of cases; five-shot GPT-4 was preferred over zero-shot GPT-4 in 64%. Despite this preference, five-shot GPT-3 scored 5.07 BLEU-4 versus 4.99 for five-shot GPT-4, showing that word-overlap metrics did not track the human preference.Source: §3.2 · pp. 7–8

3.3 Error Analysis for Matching

  • Matching errors clustered by contest for both fine-tuned CLIP and GPT-3 (p < .05), but contest difficulty was hard to predict from visual or linguistic features.The models’ per-contest accuracies were moderately correlated (ρ = .28, p ≪.001); when their predictions agreed, they were correct 87% of the time, while CLIP was correct only 38% of the time when GPT-3 was wrong.Source: §3.3 · p. 8

4 Related Work

The related work situates New Yorker caption humor in theories of incongruity and prior contest research, then distinguishes this paper’s tasks, data, and audience preferences from earlier efforts. It also connects the work to broader computational humor and joke-explanation research.

Humor

  • Humor theories group explanations around hostility, release of constraint, and incongruity; Shahaf et al. (2015) note that most contest cartoons involve incongruous situations.Source: §4 · p. 8

NLP + The Caption Contest

  • Prior contest studies analyze 5, 16, and 50 contests; this work extends them with two tasks, new resources for ranking pairs, and editor-versus-crowd preferences.Its location and uncanny-description annotations parallel Shahaf et al.’s (2015) context and anomaly tags, and it incorporates data released by earlier researchers.Source: §4 · pp. 8–9

Measuring preferences over captions

  • Caption-preference research models average rater preferences; Tanczos et al. (2017) rank captions as a multi-armed bandit problem, while The New Yorker uses NEXT, and this work draws data from Jain et al. (2020).NEXT does not directly use cartoon or contest content.Source: §4 · p. 9

Multimodal and computational humor; Explaining humor

  • Related computational humor work spans image recognition, laughter prediction, political cartoons, figurative-language inference, humor detection, and generation; joke-explanation research includes qualitative study of non-visual jokes.Chowdhery et al. (2022) qualitatively explore non-visual joke explanations; Tan (2022) characterizes explanations as identifying mechanisms behind predicted labels.Source: §4 · p. 9

5 Conclusion

Today’s vision and language models remain less effective than humans at recognizing caption relevance, reproducing crowdsourced rankings, and explaining The New Yorker Caption Contest. Their partial capabilities may still support creative collaboration, though whether the humor-understanding gap will close remains unknown.

5 Conclusion

  • Today’s vision and language models cannot match humans at recognizing caption relevance, reproducing crowdsourced rankings, or explaining The New Yorker Caption Contest.The authors describe AI’s capacity as substantial but still below human effectiveness on these tasks.Source: §5 · p. 9
  • Models may serve as creative collaborators through submission feedback, corpus-based generation, or future RLHF loops for humorous caption generation.Matching and quality-ranking models could give entrants quantitative feedback; the annotated corpus and explanations could support generation, while a future RLHF loop could operationalize model feedback.Source: §5 · p. 9
  • The authors cannot say whether the human-machine humor-understanding gap will close sooner or later.They encourage further research on the topic.Source: §5 · p. 9

6 Limitations · A Crowdworking Details

The study’s findings are limited by the contest’s cultural and stylistic scope, the subjectivity and possible bias of its preference judgments, and the single-author origin of most explanations. Its annotations and pairwise explanation judgments were gathered through separately qualified Mechanical Turk studies.

6 Limitations

  • The contest reflects a particular language, region, history, culture, style, and set of conventions, so the study does not cover all types of humor.Source: §6 · p. 9
  • New Yorker editorial selections are not ground truth for funniness; the tasks predict average preferences that may favor traits of prior winners and published cartoons.The quality-ranking framing may appear to prescribe that one joke is objectively better, although disagreement about humor is expected and valid.Source: §6 · p. 9
  • Most corpus explanations were written by one author, and future work could examine person-to-person variation despite crowdworkers rating them above machine generations.Source: §6 · p. 9

A Crowdworking Details

  • Three Mechanical Turk interfaces gathered cartoon descriptions, cartoon Wikipedia links, and pairwise explanations; the respective HITs ran 3 times per cartoon, 2 times per cartoon, and 2.7K times.Source: A Crowdworking Details · p. 13
  • For description/link annotations, 30 annotators were manually graded and 21 qualified; the study gathered descriptions and location/entity links from qualified workers.Workers were recruited from AU, CA, NZ, GB, and US, required 10K prior approved HITs and at least a 97% acceptance rate, and received a mature-content warning; annotations were collected in mid-to-late 2021.Source: A Crowdworking Details · p. 13
  • About 9 months later, 29 of 51 annotators qualified for pairwise explanation judgments after manual review of their ratings of explanations selected to span quality axes.Source: A Crowdworking Details · p. 13
  • The authors state that the study qualifies as exempt under US federal regulation 45 CFR 46 and that released crowdworker IDs were hashed to prevent annotations being traced to individuals.Source: A Crowdworking Details · p. 13

B Additional Experimental Details … B.7 GPT 3.5/GPT-4 Details

The appendix details annotation sampling and the training and prompting setups used across the experiments. It also records the constraints and costs of the GPT fine-tuning and GPT-3.5/GPT-4 experiments.

B.1 From Description details

  • For each cartoon, training used all annotations, while evaluation used one randomly sampled valid tuple fixed across methods.Tuples preserved annotator consistency across related annotation types, yielding six possible combinations rather than 36.Source: §B.1 · p. 13

B.2 CLIP

  • CLIP fine-tuning used 200-step linear warmup, a per-split validation learning-rate search, and image augmentations.Images were resized and padded to fit a 336px square input; augmentations included random horizontal flips, color jitter, and grayscaling. The Cartoon Description HIT gathered a final “Which question?” annotation that experiments did not use.Source: §B.2 · p. 13, Figure 6 · p. 14

B.3 OFA

  • OFA was fine-tuned per cross-validation split with validation early stopping, beam search (n=5), and prompts tailored to four annotation types.Training used label-adjusted smoothed cross entropy with smoothing 0.1, a maximum of 7 epochs, and a 6% warmup ratio, following the OFA authors’ method17. Direct OFA fine-tuning for explanations yielded perplexity around 300, with difficult-to-diagnose errors.Source: §B.3 · p. 16

B.4 T5-Large/T5-11B.

  • T5-Large used per-split learning-rate search, while T5-11B used a fixed learning rate; both used batch size 64.The T5-Large search tested 1e-4, 1e-5, and 5e-5; T5-11B used 1e-5.Source: §B.4 · p. 16

B.5 GPT-3 Zero Shot/In Context

  • GPT-3 zero-shot and in-context experiments used davinci-text-002, with in-context prompts containing five random training samples.The paper provides example prompts for each zero-shot task and a full randomly selected explanation-generation prompt.Source: §B.5 · p. 16

B.6 GPT-3 Fine-tuning

  • GPT-3 fine-tuning used the OpenAI API on davinci, a 175B-parameter model, with the same cross-validation setup as other models.The API’s updated parameters and davinci version were unspecified; estimated total fine-tuning cost was $600 across three tasks and five splits.Source: §B.6 · p. 16

B.7 GPT 3.5/GPT-4 Details

  • GPT-3.5 and GPT-4 were added after acceptance using the Chat API’s structured conversational prompts and default hyperparameters.GPT-3.5 queries cost an order of magnitude less than GPT-4; GPT-4 queries cost about $4K in total.Source: §B.7 · p. 16

C Task Construction Details

The paper constructs three humor-understanding tasks from high-quality contest captions, balanced mismatches, and filtered “okay” captions. It also supplies author-written explanations for a sampled set of cartoons.

Identification of High Quality Captions

  • The corpus contains 2.7K high-quality captions across 704 contests, using official finalists and semantically diverse crowd-rated captions when available.For crowd-rated contests, the authors select three diverse captions from the top five ranked by mean score; contests have three or six high-quality captions depending on whether both sources are available.Source: C Task Construction Details · p. 16

Forming Matching Instances

  • Matching instances pair each high-quality caption with its contest and four mismatched captions, balanced so each caption is correct once and incorrect four times within cross-validation splits.Mismatches are assigned only within splits, keeping training, validation, and test captions disjoint.Source: C Task Construction Details · p. 17

Forming Quality ranking Instances

  • For quality ranking, 679 contests supply “okay” captions, selected from crowd-rated middle tertiles or filtered unrated entries after removing duplicates of high-quality captions and sampled captions.For earlier contests, text-only T5-Large models trained on 506K captions discard the lowest-scoring 25% of unrated entries; validation accuracy averages 65%, with bad-label precision-at-10/20/30 of 83/77/72.Source: C Task Construction Details · pp. 17, 20
  • For captions neither obviously bad nor among the top-quality submissions, SBERT embeddings are hierarchically clustered into 0.7 · N clusters, with one representative per cluster, removing 30% of data with close neighbors.Source: C Task Construction Details · p. 20

Explanation corpus

  • An author wrote explanations for 651 of 704 sampled high-quality captions they understood, producing a 39.3K-word corpus averaging 60 words per cartoon.Captions were sampled from New Yorker finalists when available and otherwise from high-quality crowd captions.Source: C Task Construction Details · p. 20

D Graphical version of matching and ranking results.

The section visualizes matching and ranking results from Table 2. Matching results use stacked bars to distinguish model settings and mark human results; ranking results use a scatter plot with the same conventions.

D Graphical version of matching and ranking results.

E Automatic evaluation of explanations

The section reports automatic explanation-evaluation metrics but considers the main-body human evaluations better quality measures.

E Automatic evaluation of explanations

  • Table 5 reports automatically calculated explanation metrics; perplexity is unavailable for GPT-3.5/GPT-4, and the authors consider main-body human evaluations better quality measures.Source: E Automatic evaluation of explanations · p. 21

F Machine explanations that were preferred over human ones

Machine explanations were preferred in some human-versus-model comparisons, with unanimous GPT-4 preferences examined alongside more frequent majority GPT-3 preferences.

GPT-4

  • 8/130 human-versus-GPT-4 cases were preferred to the human reference by all 3 annotators.Both explanations were topical; in some cases, the machine version was arguably better because it was more succinct or offered a more meaningful detail.Source: F Machine explanations that were… · p. 21

GPT-3

  • 16/131 human-versus-GPT-3 cases favored the machine: 15 had 2/3 annotators preferring it, and 1 had 3/3.Most explanations were reasonable; GPT-3 mentioned a pop-culture reference missed by the human explanation in one case, while the authors disagreed with annotator consensus in six cases because the machine sometimes missed the point.Source: F Machine explanations that were… · p. 21

G Aiding humor generation with system-assisted brainstorming · H Related work beyond peer reviewed AI venues · I Some of our favorite New Yorker cartoons

The appendix presents a prompt for collaborative humor brainstorming and generation, surveys contest-generation projects outside peer-reviewed NLP, and lists favorite New Yorker captions.

G Aiding humor generation with system-assisted brainstorming

  • The prompt reverse-engineers brainstorming for nine finalist captions by connecting cartoon-scene associations to the concepts underlying each joke.Inspired by Wallace (2022), the process selects 1–3 associations from a broader list of possible ideas.Source: G Aiding humor generation with… · p. 21
  • The prompt combines annotated cartoon, uncanny-scene, and entity descriptions with hand-designed concepts, selected ideas, finalist captions, and annotated explanations.The examples format these components as a staged path from visual description to joke explanation.Source: G Aiding humor generation with… · pp. 21, 25
  • Figure 14 shows 8/130 cases where 5-shot GPT-4 captions beat human-authored explanations unanimously, while Figure 15 shows 16/131 fine-tuned GPT-3 cases preferred by at least two annotators.These figures present examples from the small subsets preferred by annotators in pairwise judgments.Source: Figure 14 · p. 22, Figure 15 · p. 23
  • Figure 16 demonstrates unconditional and scene-conditioned generations; the chicken-croquet caption is offered as arguably funny, though its explanation is incorrect.The authors suggest matching and quality-ranking models could filter for scene-specific generations resembling previous finalists.Source: Figure 16 · p. 24, G Aiding humor generation with… · p. 25

H Related work beyond peer reviewed AI venues

  • Outside peer-reviewed NLP, contest-generation projects used GPT-3, topic-model-conditioned RNNs, student projects, and a Markov language model trained on winning entries.The examples include The Pudding, coolposts, LILY Lab at Yale, and The Verge.Source: H Related work beyond peer reviewed AI… · p. 25

I Some of our favorite New Yorker cartoons

  • The appendix lists favorite New Yorker captions, including jokes about quitting, complimentary bread, and wanting to be feared, loved, revered, and funny.The corresponding cartoon images are accessible through the cartoonist/author names.Source: I Some of our favorite New Yorker… · p. 25
  • Other listed favorites include lines about writing down a compliment, aliens asking about Dan, carrying something in one’s mouth, and identifying a problem.The captions are attributed to JMH, JDH, AM, and RZ, respectively.Source: I Some of our favorite New Yorker… · p. 25

Related papers

Loading 2209.06293v2…