Source-linked AI summary
A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
Zhongzhan Huang, Shanshan Zhong, Pan Zhou, Shanghua Gao, Marinka Zitnik, Liang Lin
TL;DR
Evaluating multimodal LLM creativity is difficult because creative responses are subjective, diverse, and scarce. The paper uses Oogiri and proposes LoTbench, an interactive, causality-aware evaluation framework. It reports limited but near-human MLLM creativity and a strong correlation between LoTbench and multimodal cognition results.
Problem
Assessing multimodal LLM creativity is challenging because creative responses are subjective, diverse, and data-scarce.
Source: §1 · p. 2Method
The paper uses Oogiri for creativity evaluation and introduces LoTbench, a multi-round, causality-aware benchmark that measures rounds to human-level creative responses.
Source: §1 · p. 2Results
LoTbench results show limited MLLM creativity but a gap from human creativity that is not substantial, with strong correlation to MMMU and low correlation with standard creativity evaluation.
Source: §1 · p. 2Takeaways & Limitations
The authors interpret LoTbench’s strong correlation with MMMU as aligning its creativity measurements with human cognitive theories and indicating cognition’s role in early creativity.
Source: §1 · p. 2, §8 · p. 14LoTbench’s multi-turn format may not suit all subjects, and it is currently designed primarily for general LLMs.
Source: §7.5 · p. 14
Abstract
from arXiv · showhide
Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game, a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. https://lotbench.github.io
1 INTRODUCTION
The paper frames creativity as associative, non-sequential Leap-of-Thought (LoT), a capability that is difficult to benchmark in multimodal models because creative responses are subjective, diverse, and scarce. It proposes Oogiri as an evaluation platform and LoTbench as an interactive, causality-aware method, reporting limited but near-human MLLM creativity and a strong association with multimodal cognition.
1 INTRODUCTION
- LoT describes non-sequential associative thinking that bridges seemingly unrelated concepts, contrasting with Chain-of-Thought’s sequential progression.The paper presents LoT as an important ability in creativity [20], [21].Source: §1 · p. 1, Figure 1 · p. 1
- Creative evaluation is challenging because responses are subjective, diverse, and data-scarce, particularly in multimodal settings.The paper says this limitation hinders development of methods to stimulate multimodal LLM creativity.Source: §1 · p. 2
- Oogiri prompts require surprising, humorous responses to images, text, or both, aligning with MLLM inputs and drawing on high-quality human-annotated responses.The paper presents Oogiri-GO, a multilingual, multimodal dataset of more than 130,000 samples.Source: §1 · p. 2
- LoTbench uses causal reasoning in multi-round interactions to measure the average rounds needed to reach human-level creative responses, with fewer rounds indicating higher creativity.It also visualizes innovative thought processes during interactions.Source: §1 · p. 2
- Standard ranking and selection evaluations find that even advanced models, including GPT-4 and Chain-of-Thought methods, struggle with creative humor generation.The authors identify information leakage and limited interpretability as risks of standard creativity evaluations.Source: §1 · p. 2
- Most MLLMs show limited creativity, but their gap from human creativity is not substantial, and the paper reports potential to surpass humans.LoTbench results are strongly positively correlated with MMMU and weakly connected with standard creativity evaluations.Source: §1 · p. 2
2 RELATED WORKS
Related work covers multimodal LLM creativity, computational humor, and Chain-of-Thought methods. It situates the paper among work on humor generation and reasoning-focused prompting.
Multimodal LLMs and their creativity
- Multimodal LLM research addresses reasoning [4], [5], [6] and creativity [46], [47],, including applications such as scientific discovery and creative writing,,,, [53],.The passage notes challenges in humor generation, including insufficient punchlines and limited multimodal contexts,, [65].Source: §2 · p. 2
Computational humor
3 EVALUATION PLATFORM: OOGIRI GAME
Oogiri is a multimodal humor game that offers a suitable platform for studying LLM creativity. Oogiri-GO collects these responses in a multilingual dataset with human preference annotations.
3 EVALUATION PLATFORM: OOGIRI GAME
- The paper presents Oogiri as a platform for studying multimodal creativity amid a shortage of creativity data and the prevalence of text-only lateral-thinking research.The authors identify data scarcity, costly annotation, and the limited multimodal scope of earlier work as challenges.Source: §3 · p. 2
- The figure contrasts selection and ranking with LoTbench’s multi-round generation, where fewer rounds to human-level creative responses indicate greater creativity.When responses do not approach those responses, the model rethinks and tries another round.Source: Figure 2 · p. 3
- Oogiri prompts participants to produce unexpected, humorous responses to images, text, or both, matching current MLLMs’ input-output format.Its popularity provides high-quality human-annotated responses and scoring annotations such as likes.Source: §3 · p. 3
3.1 Oogiri-GO Dataset
- Oogiri-GO contains more than 130,000 samples across English, Chinese, and Japanese, with 77.95% annotated by human preferences.It includes text-to-text, image-to-text, and image-and-text-to-text tasks; its English version does not include IT2T because of cultural preference.Source: §3.1 · p. 3, Table 1 · p. 3
- Oogiri-GO was assembled from online sources and screened by Qwen-VL and human reviewers for unsafe or offensive content.The screening addressed bias, violence, explicit content, and offensive language.Source: §3.1 · p. 3
4 STANDARD EVALUATION WITH OOGIRI GAME
The standard Oogiri evaluation uses choice and ranking questions to assess models on the Oogiri-GO test set. It evaluates choices with accuracy and rankings with top-1 accuracy and NDCG.
4 STANDARD EVALUATION WITH OOGIRI GAME
- Choice questions ask models to select leap-of-thought humor responses from options, with accuracy as the evaluation metric.The four formats are 2T1, 3T1, 4T1, and 5T2, with progressively increasing difficulty.Source: §4 · p. 3
- The four choice formats add distractors or ground-truth response variants to vary question difficulty.2T1 presents a ground-truth response and a BLIP2-generated image caption; later formats add unrelated answers, a Qwen-14B rewrite, or another ground-truth response.Source: §4 · p. 3
- Ranking questions order five candidates for test items with human preferences, and are scored using top-1 accuracy and NDCG.Source: §4 · p. 3
5 LOTBENCH WITH OOGIRI GAME
LoTbench evaluates multimodal creativity through interactive rounds toward a human-level response, addressing risks of standard evaluation. CLoTv2 supports synthesis of its test data, which is manually curated into a 106-sample benchmark.
5.1 The formulation of LoTbench
- LoTbench uses masked response completion and iterative causal checks of whether responses reach DAESO; the model can ask questions and receive yes/no answers, with periodic clues.The task fixes some text to constrain responses, helping simplify evaluation of diverse creative outputs.Source: §5.1 · pp. 4–5, Figure 3 · p. 4, §5.3 · p. 7
5 LOTBENCH WITH OOGIRI GAME
- Fewer interaction rounds to reach a human-level creative response indicate higher creativity, while the interactive process addresses standard evaluations’ risks of information leakage and limited interpretability.The authors frame creativity as measuring the cost of reaching a high-quality human-level creative response rather than recognizing one among options.Source: §5 · p. 4, §5.1 · p. 5
5.2 Tuning LLM for Data Synthesis by CLoTv2
- CLoTv2 tunes LLMs through associable instruction tuning and explorative self-refinement to prepare them to synthesize human-level creative responses for LoTbench.The synthesized responses are used to construct test data intended to mitigate information leakage.Source: §5.2 · p. 5, §5.2.2 · p. 7
5.2.1 Associable Instruction Tuning
- Associable instruction tuning trains LoRA on Oogiri-GO instructions to develop generation and discrimination skills, including selection and ranking aligned with human preferences.Generation prompts may include an optional condition; choice and ranking tasks train discriminative skills.Source: §5.2.1 · pp. 5–6
5.2.2 Explorative Self-Refinement
- Explorative self-refinement uses weakly associated conditions to prompt remote associations, then trains on generated responses to strengthen the model’s LoT ability.CLoTv2 selects conditions by similarity, excluding the most and least similar candidates and adding an empty condition; the authors say this avoids many irrelevant conditions produced by random sampling in CLoTv1.Source: §5.2.2 · pp. 6–7
5.3 The Data Construction in LoTbench
- LoTbench contains 106 manually curated HHCR samples suited to masked language modeling, reflecting scarce suitable data and the need to filter culturally or knowledge-specific examples.The authors describe the benchmark as smaller than typical LLM evaluations but high quality.Source: §5.3 · p. 7
5.4 The Details of It and Rt
LoTbench evaluates creative responses through iterative interaction and judges alternatives against human responses using causal criteria. It also elicits self-questioning and scores how quickly models reach human-level responses.
5.4 The Details of It and Rt
- Across rounds, the model receives the image and caption, then its prior responses, clues, questions, and answers as context for filling a masked key text.The generation prompt includes system, in-context example, and task-specific instructions.Source: §5.4 · p. 7
5.5.1 Criteria
- DAESO requires a response to share the reference’s creative point and functional similarity, rather than merely semantic similarity.For example, a phone and an alarm clock may be functionally similar in a walnut-cracking scenario despite semantic differences.Source: §5.5.1 · p. 8
5.5.2 Modeling
- The evaluator models reference responses and captions as causal chains, then intervenes on the reference chain by replacing its key text with the model’s response.DAESO is judged by whether the creative point remains aligned and the chains’ final nodes change little.Source: §5.5.2 · p. 8
5.5.3 Measuring DAESO by E1
- E1 uses a text-based LLM to describe and reorganize the causal chains, then judge whether the response and reference satisfy the DAESO criteria.This language-based approach addresses the difficulty of explicitly constructing chains and modeling changes to their nodes.Source: §5.5.3 · p. 9
5.6 How to ask a Question Qt and answer it by E2?
- When a response is incorrect, the tested LLM proposes a speculative question about the reference, and an independent text LLM answers Yes or No.The question is intended to help the tested model generate a more human-like creative response in a later round.Source: §5.6 · p. 9
5.7 The Creativity Score Sc
- The creativity score rewards reaching human-level responses in fewer rounds and tends toward zero when the model fails to reach that level after many rounds.The score averages across repeated tests; the paper sets βc=1.0 and αc=0.2, with three repeats per sample.Source: §5.7 · p. 9
6 EXPERIMENTS UNDER STANDARD EVALUATION
Standard evaluations compare models on Oogiri choice and ranking tasks, while LoTbench tests creativity through interaction. CLoTv2 improves LoT performance, and LoTbench finds current LLMs near human levels while exposing the cost of repeated rounds.
6.1 Evaluation by Choice and Ranking Questions
- For Qwen-VL+AITv2, accuracy improves by 7.4%, 7.6%, and 5.3% on IT2T, I2T, and T2T; Qwen-VL+CLoTv2 improves by 9.6%, 11.7%, and 9.1%, respectively.Source: §6.1 · p. 10
- CLoTv2 improves CogVLM-17B on English I2T and T2T, and significantly improves single-modal LLM accuracy on English T2T.Source: §6.1 · p. 10
- CoT-like frameworks do not enhance LoT performance, whereas CLoT consistently enhances LLMs; the authors report that prompting alone does not achieve LoT.Source: §6.1 · p. 10
6.2 Experiments under LoTbench
- Most LLMs show limited creativity in LoTbench, but their gap from average human participants is not particularly large; the authors caution that the human comparison is only a reference.The testing involved 21 human subjects, grouped into three equal score-based groups.Source: §6.2 · p. 11
- Models averaged 13.65 rounds to complete the test, while Gemini 1.5 Pro averaged 12.44 rounds; the setup’s 15-round cap accompanies forward-inference costs over 10 times higher.Source: §6.2 · p. 12
- A GPT-4o mini case study visualizes the assessment process as an interpretable view of creative generation.Source: Figure 12 · p. 11, §6.2 · p. 12
7 ANALYSIS
The analyses compare human and LLM judgments of Oogiri creativity, validate LoTbench’s evaluators, and examine how LoTbench relates to cognition. They also identify limits in human evaluation, direct LLM judging, and LoTbench’s scope.
7.1 Ohter Types of Evaluation
- Human evaluation and direct LLM judging have drawbacks that motivate LoTbench’s interactive alternative.The section frames human preference and LLM-as-a-Judge as other evaluation approaches before presenting LoTbench as necessary.Source: §7.1 · p. 12, §7.1.2 · p. 12
7.1.2 LLM-as-a-Judge for LLMs’ Creativity
- Qwen-VL+CLoTv2 makes fewer ranking errors at the highest and lowest response ranks than at middle ranks, making it viable for data filtering but not precise scoring.Its Spearman correlations with human preferences occasionally fall below zero, so it is not fully reliable for arbitrary responses.Source: Figure 14 · p. 12, §7.1.2 · p. 12
7.1.1 Human Evaluation for LLMs’ Creativity
- Human evaluation aligns judgments with human creativity, but repeated studies are costly and may be unfair when different participants assess each new model.The authors characterize human evaluation as unsustainable because it requires additional manpower.Source: §7.1.1 · p. 12
7.1.2 LLM-as-a-Judge for LLMs’ Creativity
- LoTbench estimates creativity from the average interaction cost to reach human-high-quality creative responses, using a specific LLM to retain automation and mitigate leakage while improving interpretability.Lower costs indicate higher creativity.Source: §7.1.2 · pp. 12–13
7.2 The Effectiveness of E1
- GPT-4o mini is selected as E1 because supplying Exp and C and modeling causal chains improve DAESO judgments, while the information-free setting performs nearly at chance.The validation used 43 DAESO samples; the authors also cite reduced reasoning costs as a reason for selecting GPT-4o mini.Source: §7.2 · p. 13, Figure 15 · p. 13
7.3 The Effectiveness of E2
- GPT-4o mini is selected as E2, achieving 98% accuracy on 130 validation examples while having relatively low reasoning cost.Most tested models exceeded 80% accuracy on this task.Source: §7.3 · p. 13, Figure 15 · p. 13
7.4 The Correlation with Cognition Benchmark
- LoTbench results strongly correlate with MMMU, whereas standard creativity-evaluation results have no very significant relationship with cognitive ability.The authors interpret this pattern as aligning LoTbench with human cognitive theories, while noting that standard evaluations still offer other perspectives and preliminary quantitative results.Source: §7.4 · pp. 13–14, Figure 16 · p. 13
7.5 Limitation
- LoTbench’s multi-turn format may not suit all subjects, currently targets general LLMs, and measures creativity through one definition among a field without consensus.The authors say humans may struggle with long-term interactions and CLoTv2’s context-following limitations prevent its evaluation with LoTbench.Source: §7.5 · p. 14
8 CONCLUSION
The paper studies LLM creativity and Leap-of-Thought through Oogiri, introducing LoTbench to address information leakage in current assessments. It reports limited LLM creativity, a gap from human creativity that is not significant, and a strong correlation between LoTbench and MMMU.
8 CONCLUSION
- The paper investigates LLM Leap-of-Thought through Oogiri and introduces interactive LoTbench to evaluate creativity amid information-leakage concerns.The conclusion reports that LLM creativity is limited, but the gap from human creativity is not significant.Source: §8 · p. 14
- LoTbench results strongly correlate with MMMU, and the authors interpret this as aligning with human cognitive theories and cognition’s foundational role in early creativity.They describe cognition as supporting the integration of diverse concepts.Source: §8 · p. 14
APPENDIX A: THE DETAILS OF It AND GENERATION OF Rt
The appendix gives a prompt format for completing a response blank using an image and its caption, with tips and examples. It illustrates the format with a soldier image and the completion “Tutor.”
APPENDIX A: THE DETAILS OF It AND GENERATION OF Rt
- The completion prompt supplies an image, its caption, a response containing a blank, and tips, then asks for a humorous, creative completion grounded in the input.The prompt instructs the model to use divergent thinking and avoid responses resembling the supplied wrong answers.Source: Appendix A · p. 17
- The prompt includes wrong-answer examples as exclusions and presents additional examples only as placeholders.Source: Appendix A · p. 17
- In the example, clues describe the missing word as a knowledgeable person; answers connect it to school rather than the soldier, yielding “Tutor.”The completed response begins “After the tutor read my paper...”Source: Appendix A · p. 17
APPENDIX B: THE DETAILS OF RETHINKING AND GENERATING Qt
The appendix describes prompting a model to ask a general yes-or-no question about a possible missing word, using the current image, caption, response, and tips as context.
APPENDIX B: THE DETAILS OF RETHINKING AND GENERATING Qt
- The appendix’s completion guidance treats known Q&A, descriptive clues, and wrong-answer examples as hints for divergent completion while avoiding similar answers.Source: Appendix B · p. 18
- A listed example question asks whether the possible answer is edible, illustrating the yes-or-no question format.Source: Appendix B · p. 18
- The question-generation prompt reuses the image, caption, response, and tips, then asks for one general yes-or-no question that could help generate the missing word.It instructs the model to output only general questions.Source: Appendix B · p. 18
APPENDIX C: ANSWER THE QUESTION Qt AND PROVIDE At
The appendix gives examples of answering yes/no questions about a word and instructs the model to follow their format for a new input.
- The examples pair a target word and question with a direct “Yes” or “No” answer.“Mentor” is paired with a question about soldiers and answered “No”; “Cat” is paired with an animal question and answered “Yes.”Source: Appendix C · p. 19
- The prompt instructs the model to study the examples and use their output format for the new input.Source: Appendix C · p. 19
APPENDIX D: THE PROMPT DETAILS OF DAESO
The DAESO prompt asks for analysis of a humorous caption’s entities, relationships, and causal chains, then tests whether replacing a word preserves its humor and function. It specifies a yes/no summary with an explanation and a rule for complex responses.
- The prompt asks whether replacing ⟨GTW⟩ in a humorous caption with ⟨RESPONSE⟩ preserves similar humor, creativity, and function, based on analysis of the explanation.The requested analysis covers entities, relationships, and causal chains.Source: Appendix D · p. 19
- The output format requires a Yes/No SUMMARY and an EXPLANATION analyzing the entities, relationships, causal chains, and retained humor and creativity.Source: Appendix D · p. 19
- If the response is a complex format rather than a simple phrase or sentence, the required SUMMARY is No.Source: Appendix D · p. 19