PaperlayerView exact revision on arXiv

Source-linked AI summary

Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench

Reuben Narad, Siddharth Suresh, Jiayi Chen, Pine S. L. Dysart-Bricken, Bob Mankoff, Robert Nowak, Jifan Zhang, Lalit Jain

arXiv:2507.21476v1cs.CLcs.AI

TL;DR

As STEM benchmarks approach saturation, HumorBench evaluates reasoning beyond STEM by measuring how LLMs explain cartoon-caption jokes against expert-annotated objective elements. Results show transfer from STEM reasoning to humor, while test-time scaling yields mixed outcomes and scores are subject to autograder leniency.

  • Problem

    With many STEM benchmarks nearing saturation, evaluations of non-STEM reasoning involving cultural understanding and implicit knowledge remain underexplored.

    Source: §1 · p. 1

  • Method

    HumorBench tests explanations of cartoon-caption humor against annotated objective elements connecting visuals, captions, and external knowledge.

    Source: §1 · p. 1

  • Results

    STEM reasoning performance transfers to humor, STEM-only-trained models perform well, and test-time scaling yields mixed results across models.

    Source: §1 · pp. 1–2, §5 · p. 8

  • Takeaways & Limitations

    HumorBench provides a probe of higher-level reasoning that remains unsolved, and test-time compute helps only while relevant background knowledge is present.

    Source: §5 · p. 8

    The benchmark uses textual descriptions that omit some visual-cue challenges, its mildly lenient autograder makes scores upper bounds, and 499 unique elements limit fine-grained statistical resolution.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We present HumorBench, a benchmark designed to evaluate large language models' (LLMs) ability to reason about and explain sophisticated humor in cartoon captions. As reasoning models increasingly saturate existing benchmarks in mathematics and science, novel and challenging evaluations of model intelligence beyond STEM domains are essential. Reasoning is fundamentally involved in text-based humor comprehension, requiring the identification of connections between concepts in cartoons/captions and external cultural references, wordplays, and other mechanisms. HumorBench includes approximately 300 unique cartoon-caption pairs from the New Yorker Caption Contest and Cartoonstock.com, with expert-annotated evaluation rubrics identifying essential joke elements. LLMs are evaluated based on their explanations towards the humor and abilities in identifying the joke elements. To perform well on this task, models must form and test hypotheses about associations between concepts, potentially backtracking from initial interpretations to arrive at the most plausible explanation. Our extensive benchmarking of current SOTA models reveals three key insights: (1) LLM progress on STEM reasoning transfers effectively to humor comprehension; (2) models trained exclusively on STEM reasoning data still perform well on HumorBench, demonstrating strong transferability of reasoning abilities; and (3) test-time scaling by increasing thinking token budgets yields mixed results across different models in humor reasoning.

1 Introduction

As STEM benchmarks approach saturation, HumorBench addresses the need for non-STEM reasoning evaluations by testing how models explain objective joke elements in cartoons. Benchmark results show transfer from STEM reasoning to humor, while test-time scaling has mixed effects.

1 Introduction

  • Humor comprehension draws on context, cultural knowledge, and implicit connections, making it a challenging testbed for advanced AI reasoning.Source: §1 · p. 1
  • HumorBench evaluates explanations of sophisticated cartoon-caption humor by checking for objective elements that connect visuals, captions, and external knowledge.The rubric focuses on factual connections rather than subjective appreciation.Source: §1 · p. 1, Figure 1 · p. 2
  • STEM reasoning performance correlates highly with HumorBench results, STEM-only-trained models perform well, and no current model exceeds 60% accuracy on the harder subset.The hard subset includes examples requiring multiple reasoning steps or obscure cultural knowledge.Source: §1 · pp. 1–2
  • Increasing inference-time computation does not consistently improve humor reasoning across models.Source: §1 · p. 2

1.1 Why Another LLM Humor Benchmark?

  • Earlier New Yorker Caption Contest benchmarks mix understanding intended jokes with subjective humor preferences, which can influence measured performance.Source: §1.1 · p. 2
  • HumorBench isolates objective humor comprehension and its reasoning demands, and its performance correlates with other reasoning benchmarks.Source: §1.1 · p. 2

2 Related Work

Related work spans specialized reasoning models, humor benchmarks, and rubric-guided evaluation of open-ended outputs. HumorBench combines rubric-guided judging with a focus on humor reasoning.

Reasoning-focused language models

  • Reasoning-focused LLMs use STEM-heavy training, inference-time techniques, hybrid methods, and architectural or training innovations.These approaches suggest machine reasoning advances through complementary paths rather than parameter count alone.Source: §2 · pp. 2–3

Humour benchmarks

  • Humor resources test cartoon-caption understanding, pun recognition and explanation, and humor-aware information retrieval.Existing New Yorker cartoon-caption tasks include funniness ranking, caption matching, and humor explanation.Source: §2 · p. 3

Open-ended evaluation frameworks

  • Rubric-guided LLM judging has been used to evaluate open-ended outputs, and HumorBench adopts this approach for humor reasoning.Prior examples include G-EVAL, MT-BENCH, and PAPERBENCH.Source: §2 · p. 3

3 HumorBench

HumorBench turns open-ended cartoon-caption humor explanation into an element-based evaluation task using sourced cartoons, textual descriptions, and hand-annotated rubrics. Its annotations were iteratively refined, and GPT-4o autograding achieved 92% accuracy on human-labeled explanations, with a leniency bias.

3.1 Main Benchmark Task

  • Models explain a cartoon joke in their own words rather than selecting or ranking fixed answers, and a rubric of 1–3 verifiable elements makes responses gradable.Fixed answer sets may hint at punchlines or exclude valid explanations.Source: §3.1 · p. 3

3.2 Dataset: Cartoon and Caption Sources

  • The dataset draws from finalist New Yorker Caption Contest captions and original Cartoonstock captions, paired with neutral textual descriptions of relevant visual details.The benchmark targets humor comprehension rather than visual interpretation.Source: §3.2 · p. 3

3.3 Dataset: Element Annotation

  • Each cartoon-caption pair receives one to three short, directly verifiable objective elements that capture references, wordplay, implications, or other joke mechanisms.Elements avoid bundling concepts and subjective opinions about humor.Source: §3.3 · pp. 3–4

3.4 Dataset Refinement

  • Iterative review of 655 initial annotations reduced inconsistency flags below 5%, yielding 499 unique elements; a former New Yorker cartoon editor approved all 30 reviewed annotations.Cases with autograder verdict disagreement above 30% were reviewed, refined, or removed.Source: §3.4 · p. 4

3.5 Autograder and Evaluation

  • An LLM judge checks explanations against individual elements, and evaluation outputs exceeding the response limit were truncated to the last 1,000 tokens.The autograder was GPT-4o, validated against human judgments.Source: §3.5 · p. 4
  • 92% overall accuracy was achieved by GPT-4o autograding on 300 human-labeled explanations, though false positives exceeded false negatives and scores are upper bounds.The validation found no significant advantage for GPT-4o-generated explanations over those from other models.Source: §3.5 · p. 4, Table 1 · p. 4

4 Experiments

The experiments benchmark frontier models under a shared prompt and compare HumorBench performance with other benchmarks, STEM-only reasoning models, and increased test-time compute. They also examine which humor elements remain difficult, finding that category alone does not explain the hard subset.

4 Experiments

  • For consistency, experiments used a shared prompt and scaffolding, temperature 1, and no external tool calling; parameters were kept consistent where possible.The prompt was validated across Claude 3.7 Sonnet, GPT-4o, and Gemini 2.5 Pro.Source: §4 · p. 5

4.1 Main Results

  • 87.5% accuracy: OpenAI o3 leads frontier models; Gemini 2.5 Pro, Claude 3.7 Sonnet, and DeepSeek R1 each achieve approximately 80%.Newer model versions and reasoning variants generally outperform their counterparts; higher-cost models also tend to perform better.Source: Figure 3 · p. 5, §4.1 · p. 5

4.2 Transferability of Reasoning Skills

  • HumorBench accuracy is positively associated with GPQA-Diamond, ARC-AGI, and LM Arena ELO; without o-series models, its ARC-AGI correlation is ρ = 0.943 (p = 0.005).The LM Arena correlation is ρ = 0.714; the authors interpret the benchmark relationships as evidence that STEM reasoning transfers to non-STEM reasoning.Source: Table 2 · p. 5, §4.2 · p. 6, Figure 4 · p. 6

STEM-only reasoning improves HumorBench

  • STEM-only reasoning-trained R1-Zero and Phi-4 Reasoning Plus outperform their base models, and R1-Zero performs nearly on par with DeepSeek R1.R1-Zero developed reasoning through self-play on STEM problems; Phi-4 Reasoning Plus was trained on math and coding data, and both models’ reasoning traces were length-controlled for fair comparison.Source: STEM-only reasoning improves HumorBench · p. 6, Figure 6 · p. 6

4.3 Test-Time Scaling

  • Increasing test-time reasoning generally improves Qwen Plus and o-series performance, but Claude 3.7 Sonnet performs worse when its thinking budget exceeds 1024 tokens.Most completions did not exhaust their budgets; Qwen Plus rarely used over 400 tokens even with a 2000-token budget, and the authors defer investigation of this inverse-scaling observation.Source: §4.3 · pp. 6–7, Figure 5 · p. 7, Figure 7 · p. 7

4.4 Analysis of HumorBench Hard Subset

  • HumorBench-hard comprises 100 frequently missed elements, with model pass rates ranging from 60% to 0%; wordplay is under-represented by 5.4%, while cultural and toxic or shocking humor are evenly represented.The authors conclude that humor category alone is not the primary determinant of difficulty; implicit conceptual leaps or obscure references may matter more.Source: §4.4 · p. 7, Table 3 · p. 7
  • Some hard-subset examples that are intuitive and easy for humans to recognize nevertheless usually challenge LLMs.Figure 8 highlights examples, with fuller descriptions in Table 4.Source: §4.4 · p. 7, Figure 8 · p. 8

5 Conclusion and Future Work

HumorBench results suggest STEM reasoning transfers to humor, while test-time compute helps only when relevant background knowledge is present. The authors propose extensions and position the benchmark as a probe of still-unsolved higher-level reasoning.

5 Conclusion and Future Work

  • The best model still misses over 40% of elements on the hard subset, despite strong transfer from STEM reasoning and gains from specialized reasoning variants.Test-time compute helps only up to the point where relevant background knowledge is present.Source: §5 · p. 8
  • Future work could restore cartoon images for multimodal evaluation, use element rubrics for training, and improve LLM-as-judge reliability.The proposed multimodal version would assess visual recognition alongside reasoning.Source: §5 · p. 8
  • The authors hope HumorBench will spur reasoning that bridges logical deduction and nuanced human culture.They also envision it as a springboard for funny, culturally aware AI systems.Source: §5 · p. 8

Limitations

HumorBench evaluates textual explanations based on detailed image descriptions, so it excludes visual-cue recognition. Autograder leniency, possible annotation errors, and dataset size also constrain interpretation of its scores and fine-grained analyses.

Limitations

  • 92% human agreement notwithstanding, the mildly lenient autograder makes reported scores upper bounds.The authors advise treating scores as upper bounds.Source: Limitations · p. 9
  • Textual explanations from detailed image descriptions control for pure vision problems but remove the challenge of recognizing visual cues.Source: Limitations · p. 9
  • Possible annotation errors or subjective interpretations introduce noise into model evaluation.The authors note that validation and refinement were extensive, but some errors may remain.Source: Limitations · p. 9
  • The dataset's 499 unique elements limit the statistical resolution of fine-grained analyses.Source: Limitations · p. 9

A Main Benchmark Prompt

The main benchmark prompt asks a humor expert to explain the material substance of a joke using a cartoon description and caption. It constrains the response to a short, explicitly tagged explanation.

A Main Benchmark Prompt

B Autograder Prompt

The autograder checks whether a student's answer explicitly covers an anticipated comedic element. It accepts paraphrases and extra detail, but fails omissions or contradictions and requires a fixed XML output.

B Autograder Prompt

  • The grader compares the student's answer with an anticipated answer point representing a crucial comedic device or element.Source: B Autograder Prompt · p. 12
  • An answer passes when it captures the key element, including through synonyms or paraphrase, and extra details are not penalized.Source: B Autograder Prompt · p. 12
  • An answer fails when it omits or contradicts the key comedic element.Source: B Autograder Prompt · p. 12
  • The evaluation must return exactly two XML tags: <reasoning> and <judgement>, with PASS or FAIL as the judgement.Source: B Autograder Prompt · p. 12

D Model performance on HumorBench Hard

The section reports accuracy on the 100-item HumorBench-Hard subset and presents representative examples of failures on that subset.

D Model performance on HumorBench Hard

  • Table 4 presents examples where a majority of evaluated LLMs failed to identify all required humor elements.Source: Table 4 · p. 13
  • Table 5 reports accuracy on the 100-item HumorBench-Hard subset.Source: Table 5 · p. 14
Loading 2507.21476v1…