Source-linked AI summary
Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, Robert Mankoff, Robert Nowak
TL;DR
Humor generation lacks large-scale human-preference data and reliable evaluation because humor is subjective and current AI systems trail expert humans. The paper releases a massive New Yorker caption dataset, proposes GPT4- and human-based group benchmarks, and evaluates generation and alignment methods. The results show current LLMs underperform highly ranked human captions, while humor optimization raises a humor–offensiveness tradeoff that remains unresolved.
Problem
Humor generation and evaluation lack large-scale human judgments and reliable evidence about whether LLMs can produce captions aligned with highly rated human humor.
Method
The paper constructs a multimodal preference dataset from New Yorker caption contests, introduces GPT4- and group-based evaluation benchmarks, and compares models and alignment strategies.
Results
Current LLMs can generate humorous captions but significantly underperform high-ranking human submissions in the New Yorker caption contests.
Takeaways & Limitations
The benchmark provides a broad test of humor-related capabilities and supports further study of humor generation, evaluation, and alignment.
Takeaways & Limitations
Optimizing humor may increase offensiveness and toxicity, and balancing humor with cultural sensitivities remains an open challenge.
Abstract
from arXiv · showhide
We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human ratings on more than 2.2 million captions, collected through crowdsourcing rating data for The New Yorker's weekly cartoon caption contest over the past eight years. This unique dataset supports the development and evaluation of multimodal large language models and preference-based fine-tuning algorithms for humorous caption generation. We propose novel benchmarks for judging the quality of model-generated captions, utilizing both GPT4 and human judgments to establish ranking-based evaluation strategies. Our experimental results highlight the limitations of current fine-tuning methods, such as RLHF and DPO, when applied to creative tasks. Furthermore, we demonstrate that even state-of-the-art models like GPT4 and Claude currently underperform top human contestants in generating humorous captions. As we conclude this extensive data collection effort, we release the entire preference dataset to the research community, fostering further advancements in AI humor generation and evaluation.
1 Introduction
The paper introduces a large-scale dataset and benchmark for humor generation and evaluation, using human ratings of New Yorker cartoon captions. Results show current LLMs generate humor but underperform highly ranked human submissions.
- Current LLMs can generate humorous captions but significantly underperform high-ranking human submissions, making captioning a broad integration test of cultural understanding, humor recognition, reasoning, planning, and visual analysis.The paper argues that improving performance requires advancing and integrating these capabilities.
- The dataset contains over 250 million human ratings of New Yorker cartoon captions and supports research on humor expression and perception in AI systems.It is described as the first large-scale dataset with human judgments for evaluating creative tasks.
- The paper proposes benchmark metrics using GPT4 and group-based techniques to standardize evaluation and compare human- and AI-generated humor.
- The study evaluates GPT-4, Claude, and alignment strategies including RLHF, DPO, and Best-of-N sampling for humorous caption generation.
- The authors release the dataset and code to support future research on humor generation, evaluation, and alignment.
2 Related Work
Related work situates the study within New Yorker caption-contest datasets, LLM alignment research, preference datasets, and recent investigations of humor capabilities.
- The New Yorker Caption Contest Dataset has supported online-algorithm evaluation and humor research, including matching, quality ranking, and explanation-generation tasks [22].
- LLM alignment commonly uses supervised fine-tuning followed by RLHF, in which a reward model learns human preferences and PPO optimizes the model toward that reward [14] [44].
- Existing preference datasets pair prompts with chosen and rejected responses or ratings, focusing on objectives such as helpfulness, harmlessness, and answer quality [1] [4].
- Recent humor studies report that current models remain limited in humor generation and understanding despite strengths on other tasks, including in multimodal settings [26] [65].
3 New Yorker Caption Contest
The New Yorker runs weekly caption contests whose crowdsourced ratings form a large human-preference dataset. A bandit-based system allocates ratings more often to promising captions to identify strong submissions efficiently.
- Each week, readers submit captions for an uncaptioned cartoon, while editors select the top three funniest captions; the crowdsourced system provides additional ratings.
- A UCB-variant multi-armed bandit presents higher-performing captions more frequently, improving efficiency and engagement relative to random sampling.
- The contests receive close to 780,000 ratings per week, with top-5% captions averaging 821 ratings and bottom-50% captions around 85 ratings.
- The dataset covers 365 contests, from contests 530 to 895, with cartoons, captions, and ratings for each contest.
4 HumorousAI Benchmark: Funny Cartoon Caption Generation
The benchmark compares model-generated caption groups with human submissions using overall funniness, best-pick, and diversity measures. It also evaluates SFT, RLHF, DPO, and Best-of-N strategies using preference data.
- 4.1 Task: The benchmark holds out 91 contests, generates ten captions per contest, and compares them with human groups ranked #1-10, #200-209, #1000-1009, and median.
- 4.1 Task: Evaluation measures overall group funniness, best-caption selection, and caption diversity using Average EAD and SBERT similarity.
- 4.2 Evaluation Method: Group comparisons are more reliable and accurate for language-model evaluators than pairwise comparisons, and a New Yorker cartoon expert outperforms AI and crowd-worker evaluators.
- 4.2 Evaluation Method: The paper adopts GPT4-Turbo with descriptions for Group Comparison Overall and GPT4o-vision with raw images for Group Comparison Best Pick.
- 4.3 Alignment Finetuning Methods: Best-of-N sampling leaves model weights unchanged, samples N completions, and selects the completion with the highest learned reward.
- 4.3 Alignment Finetuning Methods: SFT uses cartoon-caption pairs, while RLHF and DPO use preference pairs; RLHF learns a reward model and DPO directly tunes likelihoods to match preference orderings.
5 Experiments
Experiments compare language-only, multimodal, and fine-tuned models against human captions using group-based evaluations, and examine diversity and qualitative shortcomings. Results show persistent gaps to top human performance, with multimodal integration and RLHF presenting notable difficulties.
- MLLMs vs LLMs: Language-only models such as pretrained Mistral outperformed multimodal LLaVa, while GPT-4o without vision was preferred over GPT-4o with vision.Evaluator accuracy was 67% for GPT4-Turbo with GPT4o descriptions, versus 60.5% with cartoon images and descriptions and 61.5% with a blank image and descriptions.
- Finetuning Open Source Models: Supervised fine-tuning generally hurt humor-generation performance, whereas best-of-n sampling increased Overall Win Rate but not Best Pick Win Rate and reduced diversity.The authors attribute the SFT result partly to aligning on captions ranked within the top 1000 that were often not particularly funny.
- Finetuning Open Source Models: RLHF produced lower performance than best-of-n despite increasing reward-model scores, suggesting reward maximization did not reliably improve captions.The authors hypothesize that out-of-distribution generations and the sensitivity of humorous captions to subtle wording changes make PPO optimization difficult.
- Finetuning Open Source Models: DPO significantly increased Best Pick Win Rate over the zero-shot model, unlike RLHF, while relying only on offline preference data.The authors hypothesize that avoiding out-of-distribution sampling may help when the desired humor distribution is extremely sensitive.
- Experimental Results: Claude generations were preferred over human top-10 captions only 34% of the time by workers and 1.6% by a humor expert.The expert attributed the preference for human captions partly to their better phrasing and concision.
- Diversity: Human captions were much more diverse than model-generated captions under both Average EAD and SBERT, although commercial models generally exceeded open-source models in diversity.SFT and PPO moderately improved Mistral diversity, while DPO produced a significant increase; these observations contrast with [29] for regular text generation.
6 Future Work and Societal Impact
The paper identifies persistent gaps in creative caption generation and proposes future work on evaluation, creativity, and the humor–offensiveness tradeoff.
- Improving creativity in LLM generation: Top human-generated content still significantly outperforms state-of-the-art LLMs and MLLMs, especially under expert judgment.The authors frame this gap as relevant to creative generation tasks beyond funny captions.
- Gamified evaluation of AI generated captions by a crowd: A proposed AI-versus-human rating game would compare human and AI captions across human–human, human–AI, and AI–AI matchups.Users could submit captions, while researchers could test new models and alignment methods on new cartoons.
- Gamified evaluation of AI generated captions by a crowd: Future evaluation could use crowd-based gameplay to provide a more reliable system for rating captions on new cartoons.The envisioned setup compares captions from different sources in paired judgments.
- Humor vs offensiveness tradeoff: Optimizing humor may increase offensiveness or toxicity, making the humor–offensiveness tradeoff an important next research problem.The paper emphasizes that blurred boundaries, subjective humor, and cultural sensitivities complicate alignment with human values.
B Language Model Prompts
The appendix describes prompt-based procedures for generating cartoon descriptions and captions, then evaluating captions through pairwise comparisons against human ranking groups.
- B.1 Description Generation: GPT-4o generates canny and uncanny descriptions, locations, and entities for each cartoon using five randomly selected testing examples.The same five examples are reused for every cartoon description.
- B.1 Description Generation: The description-generation prompt asks for two description types plus the cartoon’s location and related entities.The prompt supplies a cartoon image and five demonstrations before requesting the final description set.
- B.1 Description Generation: Table 7 contains examples of the generated cartoon descriptions used in the appendix.The passage identifies the table as an example resource rather than reporting an evaluation result.
- B.2 Caption Evaluation: Generated captions are compared with human entries from four ranking groups: top10, ranks #200–#209, ranks #1000–#1009, and the contestant median.Overall Comparison uses GPT4-Turbo with Hessel et al. [22] descriptions, while Best Pick Comparison uses GPT4o-vision with raw images.
- B.2 Caption Evaluation: The appendix includes an example prompt for Overall Comparison, where a judge chooses the funnier caption from two cartoon descriptions and two captions.The procedure uses five examples before the final binary choice.
- B.2 Caption Evaluation: The Best Pick Comparison prompt presents a cartoon and two captions, then requires a final answer of either A or B after five demonstrations.This version asks the judge to identify the funniest caption using the image directly.
- B.3 Caption Generation: GPT-3.5-turbo, Claude-3-opus, and GPT-4o generate ten captions per cartoon under different image and description conditions.GPT-3.5-turbo receives descriptions only; GPT-4o has image and description-only variants, while Claude receives both.
- B.3 Caption Generation: The caption-generation prompt instructs models to act as New Yorker contestants and consider objects, people, absurdity, and imagined dialogue.These cues are intended to support funny caption generation from the cartoon context.
C.1 Human Experiment Details
The human experiments obtained informed consent under institutional IRB procedures and compensated participants at $12 per hour.
- C.1 Human Experiment Details: Participants gave informed consent under Institutional IRB compliance and received $12 per hour for their participation.The reported data-collection expenditure was about $600.
C.1.1 Human Pairwise with description generated by GPT4o-vision
Human pairwise and group evaluations compare captions using either cartoon descriptions or images, while GPT-4 ranking predictions are recalibrated to address a strong option-A bias.
- C.1.1 Human Pairwise with description generated by GPT4o-vision: Each pairwise trial presents a cartoon description and two captions, asking raters to select the funniest caption.
- C.1.2 Human Pairwise with Cartoon Image: Image-based pairwise trials use the same two-caption choice format, with the cartoon shown above the alternatives.
- C.1.3 Human Group (Overall) with description generated by GPT4o-vision: Group evaluations ask raters to choose between two caption groups, either by judging the groups directly or by selecting each group’s funniest caption before comparison.
- GPT-4 ranking predictions require calibration because chain-of-thought-free group comparisons show a strong bias toward choosing option A, even when groups are swapped.The decision threshold is adjusted using a validation set and then fixed for evaluation.
C.3 Finetuning Experiment Details
The finetuning experiments define contest-based training and test splits, train several methods with parameter-efficient updates, and document implementation and sampling procedures. Results note method-specific prompt and stopping requirements, including stronger early stopping for PPO.
- Training uses contests 530–890, with the testing and validation contests from prior work forming the test split and the remaining contests used for training.
- Models are initialized from Mistral 7B Instruct or multimodal LLaVa, updated with LoRA, and trained using caption pairs with preferred examples from the top 1000 captions.
- SFT, DPO, and PPO use separate padding tokens and caption-only SFT loss, with reported checkpoints selected by early stopping before one epoch when appropriate.
- Prompt complexity differs by method: zero-shot, SFT, and preference-learning algorithms generally use simpler prompts than the preceding method.
- The UCB procedure initializes caption counts and rewards, selects captions iteratively, observes rewards from 1 to 3, and updates empirical mean rewards.
E Additional Results
Additional experiments benchmark reward models, inspect PPO training dynamics, and document the prompts used for each algorithm. Reward-model rankings guide model selection, while PPO’s improving training metrics do not translate directly into better humorous generation.
- E Additional Results: Reward-model benchmarks identify weqweasdas/RM-Mistral-7B and Eurus-RM-7B Instruct as the top two models, with the former selected for generally better ranking accuracy across settings.
- E Additional Results: PPO training shows increasing mean reward and decreasing loss, but these improvements do not correspond to better overall humorous generation.The authors hypothesize that humor’s complexity and out-of-distribution generations during RLHF may explain the mismatch.
- E Additional Results: The best prompts are documented separately for each training algorithm.
- E Additional Results: The prompt asks the model to reason about cartoon entities, absurdity, imagined conversations, and non-obvious connections before generating captions.