PaperlayerView exact revision on arXiv

Source-linked AI summary

Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation

Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, Pan Zhou

arXiv:2312.02439v3cs.AIcs.CLcs.CV

TL;DR

The paper studies whether LLMs can perform creative, associative Leap-of-Thought reasoning rather than relying on sequential Chain-of-Thought processes. It builds the Oogiri-GO benchmark and introduces CLoT, whose instruction tuning and self-refinement improve Oogiri humor and generalize to other creative tasks.

  • Problem

    Sequential Chain-of-Thought reasoning may be poorly suited to creative problem solving that requires associative leaps and out-of-the-box ideas.

  • Method

    The paper builds the multilingual, multimodal Oogiri-GO dataset and trains CLoT with associable instruction tuning followed by explorative self-refinement.

  • Results

    CLoT improves Oogiri performance across multilingual and multimodal tasks and enhances creative abilities on cloud guessing and divergent association tasks.

  • Takeaways & Limitations

    Oogiri-based training and self-refinement provide a pathway for improving LLM creative capacities across several creative tasks.

Abstract

from arXiv · show

Chain-of-Thought (CoT) guides large language models (LLMs) to reason step-by-step, and can motivate their logical reasoning ability. While effective for logical tasks, CoT is not conducive to creative problem-solving which often requires out-of-box thoughts and is crucial for innovation advancements. In this paper, we explore the Leap-of-Thought (LoT) abilities within LLMs -- a non-sequential, creative paradigm involving strong associations and knowledge leaps. To this end, we study LLMs on the popular Oogiri game which needs participants to have good creativity and strong associative thinking for responding unexpectedly and humorously to the given image, text, or both, and thus is suitable for LoT study. Then to investigate LLMs' LoT ability in the Oogiri game, we first build a multimodal and multilingual Oogiri-GO dataset which contains over 130,000 samples from the Oogiri game, and observe the insufficient LoT ability or failures of most existing LLMs on the Oogiri game. Accordingly, we introduce a creative Leap-of-Thought (CLoT) paradigm to improve LLM's LoT ability. CLoT first formulates the Oogiri-GO dataset into LoT-oriented instruction tuning data to train pretrained LLM for achieving certain LoT humor generation and discrimination abilities. Then CLoT designs an explorative self-refinement that encourages the LLM to generate more creative LoT data via exploring parallels between seemingly unrelated concepts and selects high-quality data to train itself for self-refinement. CLoT not only excels in humor generation in the Oogiri game but also boosts creative abilities in various tasks like cloud guessing game and divergent association task. These findings advance our understanding and offer a pathway to improve LLMs' creative capacities for innovative applications across domains. The dataset, code, and models will be released online. https://zhongshsh.github.io/CLoT/.

1. Introduction

The paper contrasts sequential Chain-of-Thought reasoning with associative Leap-of-Thought creativity and studies LoT through Oogiri humor generation. It introduces CLoT, combining associable instruction tuning with explorative self-refinement, and reports improved performance across Oogiri and other creative tasks.

  • LoT uses non-sequential associations and conceptual leaps, whereas CoT builds each thought sequentially for linked logical reasoning.The paper motivates LoT as a route toward creativity and innovation beyond step-by-step reasoning.
  • Oogiri provides a testbed for LoT because it requires unexpected, humorous responses to image, text, or multimodal prompts.Its extensive online presence also supplies human-generated creative content for dataset construction.
  • Oogiri-GO contains more than 130,000 multilingual and multimodal samples, while advanced LLMs including GPT-4 and CoT still show insufficient LoT ability.Direct fine-tuning is described as insufficiently effective, motivating more efficient use of humorous knowledge.
  • CLoT first tunes models with clues or empty conditions, then uses weak associations and quality selection to generate self-refinement data.Empty conditions encourage unconstrained exploration, while weakly related concepts promote connections beyond traditional cognitive limitations.
  • CLoT improves LoT performance across Oogiri games and boosts creative abilities on cloud guessing and divergent association tasks.It outperforms vanilla and CoT-integrated models on multiple-choice and ranking evaluations.

2. Related Works

The related work situates Oogiri within multimodal creativity and computational humor research, while connecting it to the broader use of Chain-of-Thought methods in LLM reasoning.

  • Oogiri is a Japanese comedy game in which players produce surprising, humorous responses to questions or random images.Its sudden insight and associative demands make it relevant to LoT evaluation.
  • Research on multimodal LLMs increasingly examines creativity for applications including scientific discovery and creative writing.The paper places Oogiri alongside this growing creativity literature.
  • Computational humor covers humor detection, interpretation, and generation, but generation still faces weak punchlines and multimodal limitations.These challenges motivate studying humor generation with multimodal LLMs.
  • Chain-of-Thought methods encourage LLMs to reason through exemplars or prompts such as “Let’s think step by step.”The paper contrasts this sequential reasoning orientation with its focus on creative association.

3. Oogiri-GO Dataset

The Oogiri-GO dataset is a large multilingual, multimodal benchmark collected and screened to evaluate LLM leap-of-thought ability through unexpected humorous responses.

  • Oogiri-GO contains more than 130,000 English, Chinese, and Japanese samples for evaluating and improving LoT ability.The dataset is built around responses to images, text, or both.
  • The dataset’s Oogiri format targets sudden insight and strong associative thinking, making it an operational test of LoT ability.Responses are expected to be both unexpected and humorous.
  • The benchmark includes Text-to-Text, Image-to-Text, and Image-and-Text-to-Text game types.These categories correspond to text, image, or multimodal inputs.
  • Data collection combines online sourcing, multimodal-LLM safety filtering, and manual screening for bias, violence, explicit content, and offensive language.The initial screening uses Qwen-VL before manual checking of the remaining data.

4. Creative Leap-of-Thought (CLoT)

CLoT improves Leap-of-Thought through associable instruction tuning, explorative self-refinement, and inference that generates, ranks, and selects creative responses. Its training and inference procedures target both associable generation and discrimination.

  • 4. Creative Leap-of-Thought (CLoT): CLoT combines associable instruction tuning, explorative self-refinement, and inference to improve LLMs’ Leap-of-Thought ability.The framework trains on Oogiri-GO-derived instructions, generates diverse data through remote association, and uses learned ranking and selection skills at inference.
  • 4.1. Associable Instruction Tuning: Associable instruction tuning trains generation and discrimination skills with LoT-oriented templates built from Oogiri-GO data.The templates support creative generation, choice-based selection, and preference-based ranking; more than 130,000 samples become over 500,000 instructions.
  • 4.2. Explorative Self-Refinement: Weakly associated or empty conditions encourage connections between seemingly unrelated concepts while preserving freedom for creative generation.Conditions are sampled from an object-noun set or left empty, helping the model explore remote associations during response generation.
  • 4.2. Explorative Self-Refinement: Explorative self-refinement ranks generated candidates, compares the top two with ground-truth responses, and retains selected high-quality data for further training.The generated instructions are combined with vanilla instruction-tuning samples into a dataset exceeding 550,000 samples.
  • 4.3. CLoT Inference: CLoT inference generates n condition-based responses, ranks them, and selects the best response from the ranked top two rather than choosing directly from all candidates.Ranking first filters low-quality candidates because selecting directly from many options has poor accuracy.

5. Experiments

The experiments evaluate LoT through progressively difficult choice and ranking questions, human preferences, and two additional creative tasks. CLoT consistently improves creative performance, while weakly associated conditions outperform strong associations and one refinement round is generally sufficient.

  • 5.2. Evaluation by Choice and Ranking Questions: Qwen-VL+CLoT improves accuracy over Qwen-VL by 9.1%, 10.4%, and 8.2% on IT2T, I2T, and T2T, respectively, across English, Chinese, and Japanese questions.
  • 5.2. Evaluation by Choice and Ranking Questions: CLoT improves LoT performance across multimodal, non-multilingual, and single-modal LLMs, while CoT-like frameworks do not consistently enhance LoT performance.The comparison includes CogVLM, Qwen-VL, text-only LLMs, CoT, CoT-SC, and prompted LoT.
  • 5.3. Human Evaluation: Users preferentially selected CLoT responses across IT2T, I2T, and T2T tasks in 154 valid surveys.The study used six LLMs and eighteen questions across three tasks.
  • 5.4. Evaluation on Other Creative Tasks: About 8% on CGG and 5% on DAT improvements show that CLoT transfers to cloud guessing and divergent association tasks.CGG uses top-1 accuracy, while DAT uses average semantic distance.
  • 5.5. Ablation Study: Weakly associated conditions outperform strongly associated conditions because stronger clues limit response diversity.The ablation attributes the advantage to more diverse LoT responses under weak associations.
  • 5.5. Ablation Study: One self-refinement round yields promising performance, whereas additional rounds provide no significant further improvement.The condition set is not expanded in the second round, limiting further gains; expanding it is identified as future work.

6. Conclusion

The paper concludes that CLoT improves LLMs’ Leap-of-Thought ability through associable instruction tuning and explorative self-refinement, using Oogiri-GO as a multimodal creative dataset. It positions Oogiri as a suitable LoT testbed because it combines multimodal prompts, creative human responses, and visualizable outputs.

  • 6. Conclusion: CLoT combines associable instruction tuning with explorative self-refinement to improve LLMs’ Leap-of-Thought ability.The paradigm uses Oogiri-GO to train LoT-oriented generation and discrimination, then generates and selects creative data for further self-training.
  • A. Introduction of Appendix: The appendix documents CLoT’s multilingual, creative-response, dataset-construction, experimental, and self-refinement analyses.It also discusses refinement rounds, performance collapse, and broader concerns about Leap-of-Thought.
  • B.1. The Novelty of Our Paper.: Over 130,000 Oogiri samples form a large-scale multilingual and multimodal dataset for investigating creativity in LLMs.The dataset includes English, Chinese, and Japanese Oogiri data.
  • B.2. The Proposed CLoT is not Tailored for Humor Generation: The paper presents CLoT as a general LoT method rather than a technique tailored only to humor generation.Oogiri serves as the study platform, while the stated target is the broader Leap-of-Thought capability.
  • B.2. The Proposed CLoT is not Tailored for Humor Generation: Oogiri is suitable for LoT evaluation because it requires creative multimodal responses, offers substantial human-generated data, and makes LoT ability visually inspectable.These properties support exploration and measurement beyond conventional sequential reasoning tasks.

C. More Experimental Results

The supplementary experiments report that CLoT improves performance across English, Chinese, and Japanese Oogiri settings and produces diverse responses on shared prompts. Additional examples cover multiple Oogiri input types and multilingual humor generation, while the appendix also outlines dataset construction.

  • C.1. The Performance in Various Languages: CLoT enhances Qwen-VL across English, Chinese, and Japanese and surpasses other advanced LLMs across multiple metrics.Figure 11 reports the language-specific comparison, with Qwen-VL as the primary baseline.
  • C.2. The Diversity Responses of Oogiri Game by LLMs: Qwen-VL+CLoT generates diverse responses for the same Chinese Oogiri samples, indicating divergent thinking across multiple perspectives.English translations are provided, but cultural factors may prevent them from fully conveying the original Chinese meanings.
  • C.3. More Examples for Oogiri-style Humor Generation: The supplementary examples span Image&Text-to-Text, Image-to-Text, and Text-to-Text Oogiri tasks across Chinese, Japanese, and English settings.English lacks Image&Text-to-Text samples, so that task is shown only for Chinese and Japanese.
  • C.3. More Examples for Oogiri-style Humor Generation: The examples include responses from several multimodal LLMs and additional Qwen-VL+CLoT outputs for Chinese Oogiri samples.Figures 13–21 provide concrete multilingual comparisons and generated-response examples.
  • D. The Construction of Oogiri-GO dataset: The appendix describes Oogiri-GO construction through online collection followed by machine and manual screening.The construction section covers the data-collection process and both filtering stages.

D.1. Online Data Collection

Oogiri-GO is collected from dedicated and general online platforms, with Bokete used as the main case study for crawling questions, answers, ratings, and metadata. The raw data is then screened by an LLM and human reviewers to reduce inappropriate content and produce a higher-quality dataset.

  • D.1. Online Data Collection: The authors collect over 200,000 raw Oogiri samples from Bokete, Twitter, Weibo, and other platforms.Bokete is selected as the representative case because it has the highest data volume and user engagement among dedicated Oogiri platforms.
  • D.1. Online Data Collection: The crawling process expands question and user ID pools from homepage rankings, user histories, and linked creative responses.Question IDs are gathered first, after which responses and their ratings are collected for CLoT discrimination training.
  • D.1. Online Data Collection: Collected responses retain text, ratings, timestamps, question IDs, and image links as structured sample metadata.The code records these fields while processing answers associated with each question.
  • D.1. Online Data Collection: For platforms that combine questions and responses in one image, PaddleOCR and positional cropping separate them into a consistent Oogiri format.This processing addresses a source-format difference between Bokete and other platforms.
  • D.2. Machine Screening by LLM: Qwen-VL performs safety screening for bias, violence, explicit content, offensive language, and related NSFW labels, reducing the dataset to about 160,000 samples.The screening uses prompts that test image, text, and image-text combinations against specified labels.
  • D.3. Manual Screening: Two rounds of manual screening by different individuals further reduce inappropriate content and leave more than 130,000 samples.The iterative design aims to reduce subjective judgment and improve screening consistency.

E.1. The Details of Implementation Details

The appendix details evaluation metrics, instruction construction, LoRA tuning, weak-association extraction, and user-study procedures for CLoT experiments.

  • E.1. The Details of Implementation Details: Choice accuracy, NDCG with top-1 accuracy, vote proportions, and ASD evaluate Oogiri, user-preference, CGG, and DAT experiments.ASD averages semantic distances among the ten words following each choice-question completion.
  • E.1. The Details of Implementation Details: Explorative self-refinement samples five candidates under empty or noun-based weakly-associated conditions, ranks them, then selects from the top two and ground-truth responses.Empty conditions occur with probability 0.5, while noun conditions encourage links between different concepts.
  • E.2. The Details of Instruction Templates: Instruction templates convert Oogiri data into associable generation and discrimination tasks using task-specific prompts, responses, conditions, options, and multimodal placeholders.IT2T data scarcity motivates masked-language-modeling augmentation from I2T and T2T answers.
  • E.2. The Details of Instruction Templates: Three original prompt templates are used per task because experiments found no significant performance difference from expanding Image-to-Text templates to fifty.Image&Text-to-Text templates additionally use a [MASK] character.
  • E.3. The Details of Tuning by LoRA: LoRA is trained on associable instruction data while the original LLM weights remain frozen, and textual-only insertion achieves 38.8 accuracy versus 37.0 for both modules and 25.4 for visual-only insertion.These results determine the textual-module strategy for Qwen-VL.
  • E.4. The Extraction of Weakly-associated Conditions: Weakly-associated conditions are built by extracting, deduplicating, and manually screening nouns from multilingual Oogiri responses using NLTK, Jieba, and Janome.The resulting noun set S supplies conditions for explorative remote association.
  • E.5. The Details of User Study: The user study presents eighteen questions across IT2T, I2T, and T2T, randomly permutes six LLM responses, and aggregates selections from 154 valid questionnaires totaling 2772 votes.Users choose the response they consider most creative and humorous.

F. The Details of other Creative Tasks

The paper evaluates CLoT beyond Oogiri using cloud-shape classification and divergent association, while also motivating a one-round self-refinement analysis.

  • F.1. The Details of Cloud Guessing Game (CGG): CGG uses four visually distinct categories, manually screened generated cloud images, and four-option questions containing one category label plus unrelated distractors.ControlNet scale adjusts image difficulty by controlling mask correlation.
  • F.1. The Details of Cloud Guessing Game (CGG): CLoT further enhances Qwen-VL performance on the Cloud Guessing Game, where models classify white-cloud images by selecting the resembling shape.The task uses classification accuracy and is intended to probe associative LoT ability.
  • F.2. The Details of Divergent Association Task (DAT): DAT adapts a classic creativity test into nine-option questions requiring selection of the word most semantically distant from a given set, measured by GloVe-based ASD.The benchmark targets associative ability through divergent word selection.
  • G. The Analysis for Self-Refinement: The appendix frames one-round self-refinement as an analysis target because additional rounds were examined for their effect on LoT ability.The subsequent discussion considers why repeated refinement does not substantially improve performance.

G.1. The Discussion for the Round of Self-Refinement

Additional self-refinement rounds do not significantly improve LoT because available diversity is largely exhausted after one round, while CLoT avoids performance collapse through diverse conditions and filtering.

  • G.1. The Discussion for the Round of Self-Refinement: A diverse random subset outperforms an equally sized animal-only subset, showing that creative-data diversity matters for generating effective responses under weak associations.The comparison uses two 10,000-sample Oogiri-GO subsets.
  • G.1. The Discussion for the Round of Self-Refinement: Diversity in noun set S is also important, but expanding it is constrained by limited response nouns and the difficulty of finding external conditions that are weakly related rather than irrelevant.These constraints limit the quality and variety of newly generated conditions.
  • G.1. The Discussion for the Round of Self-Refinement: Additional self-refinement rounds do not significantly enhance LoT, so CLoT defaults to one round that effectively uses existing diversity in creative data and noun set S.One round already achieves satisfactory performance under these diversity constraints.
  • G.2. Self-Refinement doesn’t Cause a Performance Collapse in CLoT: CLoT’s explorative self-refinement does not cause performance collapse because weakly-associated conditions diversify generated data and tuned discrimination filters it for quality.Main-text experiments report no performance impact and improved LoT ability.

H.1. The Oogiri-GO Dataset doesn’t Include English IT2T Type Data

Oogiri-GO excludes English IT2T data because this format is culturally less common and substantially harder to create, although other data types remain available for CLoT validation.

  • H.1. The Oogiri-GO Dataset doesn’t Include English IT2T Type Data: English IT2T data is absent from Oogiri-GO because English-speaking communities favor I2T or T2T formats and IT2T content requires OCR, image editing, and translation.These cultural and production barriers make English IT2T data scarce online.
  • H.1. The Oogiri-GO Dataset doesn’t Include English IT2T Type Data: The dataset’s lack of English IT2T samples is a stated scope limitation, while its ample other data types support validation and analysis of CLoT.The limitation concerns English IT2T coverage rather than Oogiri-GO overall.

H.2. Fine-tuning Directly on Oogiri-GO is Hard to Achieve Good LoT Ability

Direct fine-tuning on Oogiri-GO produces a substantial performance gap relative to CLoT because it lacks targeted associative exercises and cannot reliably exploit uneven creative quality. The paper also identifies prompting, knowledge retention, and human-feedback evaluation as directions for further improving CLoT.

  • H.2. Fine-tuning Directly on Oogiri-GO is Hard to Achieve Good LoT Ability: Direct fine-tuning on Oogiri-GO shows a significant performance gap compared with CLoT.The comparison is presented for the 3T1 and I2T settings.
  • H.2. Fine-tuning Directly on Oogiri-GO is Hard to Achieve Good LoT Ability: Direct fine-tuning mainly fits observed creative patterns rather than stimulating novel associative thinking, especially when creative data are scarce.The authors argue that dataset fitting can trap the model in local patterns and therefore requires targeted associative exercises.
  • H.2. Fine-tuning Directly on Oogiri-GO is Hard to Achieve Good LoT Ability: Uneven response quality limits direct fine-tuning because mundane examples are insufficient to foster strong LoT abilities.Only exceptionally imaginative instances are described as having the potential to stimulate LoT.
  • H.2. Fine-tuning Directly on Oogiri-GO is Hard to Achieve Good LoT Ability: CLoT combines associative exercises with discrimination of creative-data quality, yielding performance beyond direct fine-tuning.This contrast motivates the two-stage design discussed in the appendix.
  • H.3. How to Further Enhance CLoT?: Future CLoT improvements could use human rankings to construct evaluators for reinforcement learning from human feedback, while preserving pretrained knowledge through continuous learning.The paper also reports that prompting alone is insufficient to stimulate LoT and that instruction tuning can cause partial forgetting.
Loading 2312.02439v3…