ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models

• Humor supports relationships, motivation, emotion, attention, and potentially stronger human-computer interaction, yet computational humor remains far from human-like and is considered AI-complete.Source: §1 · p. 1
• ChatGPT can converse contextually and tell jokes, but repeated, unusually polished outputs suggested that many jokes might be reproduced from training data or a predefined list.Source: §1 · pp. 1–2
• The study uses structured prompt-based experiments because ChatGPT’s internal workings are inaccessible, examining joke generation, explanation, and detection.Source: §1 · p. 2
• The paper cautions that human-like terms such as “understands” and “explains” describe behavior by analogy, not evidence that AI reaches a human or conscious level.Source: §1 · p. 2

• Transformer-based LLMs such as BERT and GPT capture contextual relations and implicitly included information, making them relevant to computational humor.Source: §2 · p. 2
• ChatGPT is a conversational GPT3.5-based model fine-tuned with reinforcement learning from human feedback, but it can still produce factually wrong information.Source: §2 · p. 2
• Computational humor mainly studies humor detection or classification and humor generation, while researchers emphasize automated humor’s complexity and current limitations.Source: §2 · p. 2
• Although humor theories are often logic-based, artificial-agent research is predominantly data-driven, with pretrained language models playing a central role.Source: §2 · p. 2

3 Method

• The experiments ran in fresh conversations through a wrapper because the ChatGPT web service lacked an official API and limited prompts to 74 per hour.Source: §3 · pp. 2–3
• The study distinguishes originally generated text, exact replicated text memorized from training data, and modified output combining memorized material with alterations.Source: §3 · p. 3

3.1 Joke Generation

• 3.1 Joke Generation: ChatGPT received 1,000 joke requests using ten differently worded prompts, and the researchers identified 25 recurring top jokes for later tasks.Source: §3.1 · p. 3

3.2 Joke Explanation

• 3.2 Joke Explanation: The model was asked why each top joke was funny to test whether generation ability extended to understanding humor’s underlying mechanisms.Source: §3.2 · p. 3

3.3 Joke Detection

• 3.3 Joke Detection: Researchers modified the 25 top jokes by removing wordplay, topic, structure, or combinations of these cues, then asked ChatGPT to classify each sample.Source: §3.3 · pp. 3–4
• Across generated samples, nearly all jokes used a question-answer structure and wordplay puns, while joke topics often involved bizarre scenarios or personifications.Source: §3.3 · pp. 3–4

4.1 Joke Generation

• Among 1008 samples, 909 were identical to one of 25 existing jokes, indicating replication rather than original generation, while the remaining samples were not necessarily new.About half of the remaining 99 samples modified the top jokes; some still made sense, whereas others lost their conclusiveness.Source: (1.2) Why didn’t the bicycle stand up… · p. 5
• Of 1008 samples, 909 were identical to one of the top 25 jokes, and all 25 puns together accounted for about 90% of the gathered puns.The top four jokes occurred over a hundred times each and made up more than 50% of the samples, indicating limited versatility in ChatGPT’s response pattern.Source: (1.2) Why didn’t the bicycle stand up… · p. 5
• Prompted examples show ChatGPT recombining joke elements and adapting templates to specific contexts, but increasingly unfamiliar content often lacks a convincing comic element.Some modifications preserve wordplay and validity, whereas others become invalid or merely template-like.Source: (2.5) Why did the duck cross the… · p. 5, (4.4) Why was the physics book sad?… · p. 6, (5.1) Why did the cat put his paw in… · p. 6

4.2 Joke Explanation

• For 23 of 25 valid jokes, ChatGPT’s explanations were accurate and reasonable, correctly covering direct and acoustic double meanings, including “too tired” and “two-tired” or “peeling” and “feeling.”It correctly explained the double meaning of “problems” and the chicken anti-joke.Source: §4.2 · pp. 6–7
• ChatGPT also generated convincing but fictional explanations for invalid or unsuitable joke-like samples instead of acknowledging that they lacked humor.It applied an approved reasoning scheme to inappropriate cases and invented wordplay that made no sense from a human perspective.Source: (9.1) The humor in this joke lies in… · p. 7

4.3 Joke Detection

• The study therefore tested joke detection by asking ChatGPT to classify five conditions for each joke, including valid jokes themselves.This experiment directly examined whether the system could distinguish jokes from unfunny but joke-like samples.Source: §4.3 · p. 7
• ChatGPT’s joke-detection responses fell into three categories: explicit joke or pun recognition, secondary mention of humorous potential, and no joke classification.The responses also contained diverse details about sentence structure and stylistic elements.Source: §4.3 · p. 7
• Figure 2 presents the modification steps and overall results of the joke-detection task.Detailed category definitions, modifications, and classifications are provided in Appendix Sec. D.Source: §4.3 · p. 7
• All 25 original jokes were clearly classified as jokes, while modification sets B and D were entirely classified as no jokes.Sets A and C, retaining two characteristics, produced mixed classifications with a slight tendency toward no joke and few potentially humorous judgments.Source: §4.3 · pp. 7–8
• Samples with one joke characteristic were not sufficient for joke classification, whereas samples with two characteristics received mixed classifications and all samples with three were classified as jokes.This pattern applied to both joke-typical structure and joke-typical topic.Source: §4.3 · p. 8

Joke Generation

• More than 90% of generated samples were the same 25 jokes, indicating recurrence rather than predominantly original generation.The authors hypothesize that frequent instances may be learned or memorized from training data, but cannot test this without access to training details.Source: §5 · p. 8
• Modified or generated jokes and uneven output frequencies do not support hard-coded responses, but reveal a one-sided conception of humor based on learned structures and wordplay.ChatGPT can fit new topics into known joke patterns while failing to reflect a broader spectrum of humor.Source: §5 · p. 8

Joke Explanation

• ChatGPT explains personifications and wordplay well, yet invents convincing explanations instead of acknowledging when an invalid joke lacks a valid explanation.It struggles when sequences do not fit learned patterns.Source: §5 · p. 8

Joke Detection

• A single cue such as a question-answer template does not reliably mislead ChatGPT, whereas samples with more joke characteristics are more likely to be classified as jokes.The tested characteristics were structure, wordplay, and topic.Source: §5 · p. 8

Experimental Scope

• The experiments used empty, refreshed chats, although humor perception depends importantly on conversational context that ChatGPT can capture and use.The authors identify contextual investigation as future work.Source: §5 · p. 8

6 Conclusion

• ChatGPT correctly identifies, reproduces, and explains puns that fit learned patterns, but fails on other kinds of puns and cannot confidently create intentionally funny original content.The conclusion summarizes the results across the three prompt-based tasks.Source: §6 · p. 8
• The observations indicate that ChatGPT learned a specific joke pattern rather than becoming broadly funny, while focusing on content and meaning more than superficial characteristics.The authors describe this as a large leap toward general humor understanding compared with previous LLMs.Source: §6 · pp. 8–9

Future Work

• The authors plan comparisons with GPT4 and open-source models such as LLaMa and GPT-NeoX on joke generation and understanding.These comparisons are proposed future work.Source: §6 · p. 9

Limitations

• Humor is subjective and diverse, while this investigation focuses only on standalone jokes rather than other manifestations requiring a more complex setup.Odd or anti-jokes may be judged differently by ChatGPT and humans, complicating evaluation.Source: Limitations · p. 9
• The authors cannot confidently map system outputs to specific input data because the model, training data, and RLHF examples are inaccessible.The prompt-based observations provide intuition about capabilities and limitations, but further insight is needed.Source: Limitations · p. 9

Ethics Statement

• ChatGPT’s broad societal impact carries responsibilities and risks including discriminatory biases and misinformation.Source: Ethics Statement · p. 9
• The system does not meet open-science requirements because its data, training details, and model characteristics are private.The authors frame their work as contributing to objective understanding of ChatGPT’s capabilities and limitations.Source: Ethics Statement · p. 9

A Access Dates

• Observations may change across model updates, so the authors report experimental access dates for joke generation, explanation, and detection.The listed experiments occurred during January–March 2023.Source: A Access Dates · p. 11

B Joke Generation - Top 25 Jokes

• The majority of generated samples were the same 25 puns, including a scarecrow joke occurring 140 times and a tomato joke occurring 122 times.Source: B Joke Generation - Top 25 Jokes · p. 11

C Joke Explanation

• 23 of 25 generated explanations were valid, with only E8 and E16 judged inaccurate.Source: C Joke Explanation · p. 11
• ChatGPT explains wordplay by identifying double meanings, homophones, incongruity, and unexpected punchlines.Source: C Joke Explanation · p. 12

D Joke Detection

• The detection task compares original jokes with variants that remove wordplay, topic, structure, or structure and wordplay.Source: D Joke Detection · p. 13
• ChatGPT groups classification responses into “Joke or pun,” “potentially humorous,” and “no joke,” based on whether comic qualities are identified.Source: D Joke Detection · p. 13

• Removing wordplay is intended to eliminate the jokes’ comic element while minimally altering their wording.Source: §D.1 · p. 13
• The resulting samples preserve recognizable question-and-answer setups but replace humorous punchlines with literal responses.Source: §D.1 · p. 13
• Table 1 organizes the detection experiment by condition, included characteristics, sample count, and the three response classes [P], [H], and [N].Source: Table 1 · p. 14, §D.1 · p. 14

• Modification B replaces joke-typical protagonists or topics with ordinary human or neutral alternatives while retaining the question-answer format.Source: §D.2 · p. 14
• Because the topic change removes the original wordplay, the resulting samples are designed to lose their humor despite retaining familiar sentence patterns.Source: §D.2 · p. 14
• Examples replace humorous objects or characters with people, such as changing a tomato, computer, or scarecrow into a human protagonist.Source: §D.2 · p. 14
• The examples include explanations that a woman went to the doctor because she was feeling crumbly, a man could not stand because he was drunk, and a student was cold because it was winter.Source: §D.2 · p. 14

D.3 Modification C - Minus Structure

• Modification C changes only sentence structure, leaving the original joke topic and pun intact.Source: §D.3 · p. 15
• Across the examples, the modifications separate the contributions of structure, topic, and wordplay to the original jokes.Source: §D.3 · p. 15, §D.4 · p. 15
• The structure-removed samples recast jokes as ordinary declarative sentences, such as “The chicken crossed the playground to get to” or “The computer was cold because it left its Win-”.Source: §D.3 · p. 15

D.4 Modification D - Minus Wordplay 2

• Modification D combines structure removal with wordplay removal, producing potentially unfunny one-line statements.Source: §D.4 · p. 15
• Examples of Modification D replace puns with literal explanations, including hiding money from intruders, making a smoothie, or having a scratch in a car.Source: §D.4 · p. 16