Source-linked AI summary
CREATE: Testing LLMs for Associative Creativity
Manya Wadhwa, Tiasa Singha Roy, Harvey Lederman, Junyi Jessy Li, Greg Durrett
TL;DR
Evaluating associative creativity is difficult because real-world creative outputs are subjective, while abstract symbolic tests may not reflect practical use. CREATE addresses this gap with a benchmark of real-concept graph paths scored for factuality, specificity, and diversity; frontier models achieve higher creative utility, although model coverage remains incomplete and more reasoning effort is not always better.
Problem
Existing creativity evaluations are either subjective on complex real-world tasks or potentially unrepresentative when they use abstract symbolic tasks.
Method
CREATE asks LLMs to generate sets of real-world concept connections and scores them for factuality, specificity, and diversity using graph paths and creative utility.
Results
Frontier models achieve higher creative utility than weaker or open-source models, but no single model adequately covers strong answer paths and more reasoning tokens do not necessarily improve scores.
Takeaways & Limitations
CREATE provides an evaluatable benchmark and sandbox for studying and improving LLMs’ associative creativity.
Takeaways & Limitations
Creative-utility validation relies on subjective author judgments and annotators’ non-uniform domain knowledge, while standard LLM use typically produces only moderate-sized output sets.
Abstract
from arXiv · showhide
A key component of creativity is associative reasoning: the ability to draw novel yet meaningful connections between concepts. We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning. CREATE requires models to generate sets of paths connecting concepts in a model's parametric knowledge. Paths should have high specificity (distinctiveness and closeness of the concept connection) and high diversity (dissimilarity from other paths), and models are scored more highly if they produce a larger set of strong, diverse paths. This task shares demands of real creativity tasks like hypothesis generation, including an extremely large search space, but enables collection of a sizable benchmark with objective answer grading. Evaluation of frontier models shows that the strongest models achieve higher creative utility than others, with the high multiplicity of answers and complexity of the search making benchmark saturation difficult to achieve. Furthermore, our results illustrate that thinking models are not always more effective on our task, even with high token budgets. Recent approaches for creative prompting give some but limited additional improvement. CREATE provides a sandbox for developing new methods to improve models' capacity for associative creativity.
1 Introduction
CREATE addresses the difficulty of evaluating associative creativity by testing whether LLMs can generate multiple strong, diverse connections between real-world concepts. Its results show that frontier models score higher, but no single model covers the space of strong paths and more reasoning tokens do not necessarily improve performance.
- Creativity is central to scientific discovery, writing, and creative problem solving, but complex real-world queries are difficult and subjective to evaluate.
- CREATE benchmarks associative creativity by asking LLMs to form open-ended associations among real-world entities using world knowledge and multi-hop reasoning.The benchmark balances real-world applicability with verifiability.
- CREATE queries span domains such as movies, positions of power, and genes associated with diseases, requiring models to enumerate open-ended, trivia-like connections.
- The benchmark’s creative utility metric rewards connections that are both strong and distinct rather than simple variations on one theme.
- Frontier models achieve higher creativity utility than weaker or open-source models, but no single model adequately covers strong answer paths and more reasoning tokens do not necessarily raise scores.
2 Background
Prior creativity tests challenge humans but are comparatively easy for LLMs, while specialized model benchmarks face difficulties in evaluating quality, diversity, and usefulness. Homogeneous outputs further suggest that current models may face a creativity ceiling.
- Human creativity tasks tax working memory and coordination of multiple concepts, but are comparatively easy for LLMs with large contexts, retrieval, and parallel generation.
- Specialized creativity benchmarks for writing, scientific ideation, and code generation are difficult to evaluate at scale because LLM judges can conflate quality, diversity, and usefulness.
- Models are criticized for returning homogeneous outputs, indicating a possible performance ceiling even when systems are ensembled.
3 Conceptual Framework
The framework treats associative creativity as selecting a set of graph paths that balance individual quality with diversity. CREATE evaluates paths for structural validity, factuality, specificity, and distance from other paths.
- The framework defines a task’s possible outputs as U(x), while an LLM samples a moderate-sized subset U whose high-quality items may be difficult to find.
- Creative utility combines item quality, pairwise distance, and a patience parameter that controls how strongly larger output sets are favored.
- The framework computes an ordering greedily because LLM outputs are treated as unordered sets but creative utility requires an ordering.
- Associative creativity on graphs: Associative creativity is instantiated as finding graph paths that link concepts through structured relationships, with potentially many valid paths and a right-tailed quality distribution.
- CREATE Formalization: A path is structurally valid when consecutive triples share entities, and factual when its triples represent true relations.
- Quality of Paths: Path quality requires factuality and specificity, with specificity determined by the weakest triple and lower scores for generic relations.Specificity reflects how many entities could plausibly participate in a connection.
- Specificity and factuality are implemented with LLM judges, while path distance is computed from transformed cosine distance between path-string embeddings.
4 Constructing the CREATE Benchmark
CREATE constructs natural-language queries from Wikidata graph structures designed to contain at least one strong connection, then evaluates models on generating multiple distinct and strong paths rather than reproducing a reference path.
- The dataset begins with manually selected Wikidata relation–category pairs whose classes are compact and semantically coherent.
- Unordered entity pairs within each class are expanded with an informative outgoing edge to create multi-hop source paths.
- GPT-4o-mini rewrites structured triples into natural-language queries, such as asking for a connection between Sebastian Vettel and someone partnered with Kelly Piquet.
- The source path establishes that at least one strong connection exists, but models are evaluated on producing multiple distinct and strong paths rather than matching it.
- The benchmark contains 931 natural-language queries spanning eleven relations and starting entities involving people, genes, and chemical compounds.
- Triple specificity and factuality are estimated with gpt-oss-120b judges, while path distance uses all-MiniLM-L6-v2 embeddings and transformed cosine distance.
5 Experimental Setup
The experiments compare thinking and non-thinking language models under a shared prompting setup, while testing several interventions intended to elicit more diverse and creative relational paths.
- Models: The evaluation covers non-thinking and thinking large language models spanning architectures, sizes, and training paradigms.
- Base prompt: All models receive a shared base prompt instructing them to generate multiple high-quality relational connections as structured JSON.
- Prompt variations: The study examines whether alternative prompts can elicit more diverse and higher-quality relational paths.
- Prompt variations: The prompt variations include creativity instructions, verbalized sampling, iterative regeneration, and independent resampling.
6 Metric Validation
The validation studies test whether CREATE’s component metrics and aggregate creative utility align with human judgments and produce stable model rankings. Qualitative and figure-based analyses further illustrate high-scoring, model-specific paths and benchmark performance patterns.
- Validation design: Human validation experiments assess specificity, factuality, and creativity of generated paths.
- Specificity: Specificity scores achieve substantial annotator agreement, with Krippendorff’s alpha of 0.68.
- Factuality: Factuality judging reaches 85.9% balanced accuracy on 346 human-labeled triples, with high recall for incorrect relations and high precision for correct ones.
- Judge robustness: Changing the judge model leaves model rankings unchanged, supporting robustness of the specificity and factuality evaluations.
- Creative utility: Creative utility correlates with human judgments, while direct end-to-end LLM scoring is less stable than the decomposed metric.Annotator agreement is moderate at Krippendorff’s alpha 0.62, and correlations between annotator averages and the metric reach 0.55.
- Qualitative evaluation: Qualitative examples show high-scoring paths that are unique to individual models, indicating that different models surface different interesting connections.
- Performance analyses: Frontier-model comparisons report creative utility against patience, reasoning effort, and prompt variations, with GPT-5.5 and Claude-Sonnet-4.6 achieving the best scores.
7 Results
Frontier models achieve the highest creative utility, but performance varies by model and additional reasoning or prompting does not consistently improve results. Iterative and resampling interventions can help, while model outputs remain broadly similar in diversity.
- Frontier models achieve the highest creative utility, with top models sustaining quality and diversity across larger answer sets.At patience 1, scores reach 18 on average, and GPT-5.5, Claude-Sonnet-4.6, and GPT-5 rank first through third across patience values.
- Some models underperform because they are conservative or generate too few paths, even under high reasoning effort.Gemini 3.1-Pro hesitates to produce potentially non-factual answers, while Claude-Opus-4.7 does not appear to think long enough to generate many paths.
- Additional reasoning has model-specific returns: GPT-5-mini benefits substantially, whereas gpt-oss-120b shows little to no improvement.The results suggest that gains depend on how effectively a model uses its reasoning trace to plan diverse, high-utility paths.
- Gemini-3-Pro can match GPT-5-mini’s utility with fewer paths by producing higher-quality and more distinct paths.The analysis reports average maximum quality, average pairwise distance, factual-path counts, and path lengths as explanatory measures.
- Prompt interventions affect creativity weakly and inconsistently, although iterative prompting increases utility and resampling performs best.Both stronger interventions use a substantially larger inference budget, while verbalized prompting produces few valid and factual paths.
- Across models and prompt variations, generated paths are largely similar, with iteration producing more distinct responses than resampling alone.A small fraction of paths nevertheless has high distance from the population and is creatively interesting.
8 Reasoning Trace Analysis
Reasoning traces reveal enumeration and backward chaining as strategies for generating connections, but repeated nodes can undermine search effectiveness. These patterns identify targets for improving search efficiency and inference-time interventions.
- Models brainstorm from both ends of a query, using enumeration and backward chaining to fill in missing details.Figure 9 highlights enumeration and backward chaining in reasoning traces.
- Models often revisit the same nodes, producing repeated information that undermines the effectiveness of the search process.Characterizing these strategies helps identify where reasoning helps or hurts creative search.
- Iterate and Resample interventions lead to the highest creative utility scores among the alternative prompting methods shown.The figure caption notes that prompting improvements depend on the model.
9 Conclusion
The paper introduces CREATE as an evaluable benchmark for associative creativity and finds that current models show useful capabilities while higher thinking effort is not necessarily sufficient. Its dataset is scoped to English, Wikidata-based knowledge, and selected domains, limiting coverage of other languages and applications.
- CREATE benchmarks associative creativity by requiring sets of strong, distinct connections between real concepts while preserving evaluability.
- The results highlight strengths of current models but show that higher thinking effort is not necessarily a silver bullet in these settings.
- The dataset is limited to English, draws on facts available on Wikidata, and covers curated domains including people, genes, and medical drugs.
- The curated domains are intended to represent trivia connections and ideation but do not perfectly cover associative creativity in other languages or domains.
B Benchmark
The benchmark constructs natural-language connection queries from knowledge-graph relations and evaluates generated paths using specificity, factuality, and diversity-oriented distance measures. Its data-generation process filters trivial or overly broad relations before creating queries.
- The data-generation process removes links where the source entity also satisfies the tail connection and excludes relations with very large class memberships.
- Algorithm 1 forms entity pairs within relation-defined classes, expands one endpoint using an informative outgoing fact, and converts the resulting structure into a natural-language query.
- The benchmark is licensed under CC BY-SA 4.0.
- Cosine distance is transformed for the creative utility metric, with distances above 0.7 indicating substantially different paths and values below 0.4 indicating similar or paraphrased paths.
- Specificity and factuality are evaluated with Prompts K.1 and K.2, while specificity uses elicited class sizes mapped to a 1–5 scale.
E.1 Factuality
The factuality evaluation uses an LLM judge validated against human annotations, while human creativity judgments assess diversity and semantic spread across generated paths. The evaluator performs reasonably well but has identifiable weaknesses on niche entities and ambiguous relations.
- Factuality: The factuality evaluator was validated on 346 annotated relations and achieved 0.86 balanced accuracy.
- Factuality: The evaluator commonly misjudges niche or long-tail entities and relations, and may assess general entities too strictly even when the relation holds.
- Factuality: The selected gpt-oss-120b evaluator performs fairly well relative to GPT-5-mini, GPT-4o-mini, and GPT-4.1-mini given cost and effectiveness.
- Creativity evaluation: Human creativity annotations use a 1–5 scale in which higher scores correspond to broader semantic coverage, salient intermediate entities, and less path clustering.
F Stability of Creative Utility vs. Stability of End-to-End Evaluation
The study tests creative utility across multiple evaluator models and finds it more stable than directly prompting LLMs for creativity judgments. It also analyzes distinctive paths and evaluates prompt and model configurations across a broad set of systems.
- Experimental setup: The evaluation compares GPT-5, Gemini-3-pro, Claude-Haiku-4.5-med, and GPT-5-mini across reasoning levels, alongside multiple evaluator models.
- Metric stability: Creative utility remains stable across different factuality and specificity evaluators, whereas direct LLM creativity judgments vary by over one point on a 1–5 scale.
- Metric stability: Agreement among evaluators is quantified with Krippendorff’s alpha across six models and 900 total instances.
- Distinctive paths: For a sample query, the analysis selects the factually correct quality-≥3 path with the highest minimum distance from the population as the most distinctive path.
- Prompting: Prompt variations include verbalized sampling, an explicit creative instruction, and an iterative prompt designed to test whether prompting influences creative responses.
- Experimental setup: Inference uses LiteLLM, proprietary-model APIs, self-hosted vLLM endpoints, and NVIDIA H200 GPUs for open-source models.
- Experimental setup: Open-source evaluations use 16k and 32k token settings where possible, with selected reasoning-effort exceptions for Claude Opus and GPT-5-mini.
J Results
Results show that model behavior varies across prompting, distinctiveness, and factuality settings, with no single strategy consistently maximizing creative utility. Analyses also reveal inefficient search traces and a tradeoff between utility and factuality.
- Prompting: Creative prompting changes relation distributions little, while verbalized sampling reduces the number of generated paths.The added probability verbalization may account for the reduction in paths.
- Model comparisons: 13 of 15 model pairs differ significantly in mean creative utility at patience=1.0 after Bonferroni correction.The two nonsignificant comparisons are Claude-Sonnet-4.6 vs. GPT-5.5 and Claude-Opus-4.7 (xhigh) vs. Gemini-3-pro.
- Distinctiveness: Distinctiveness values are similar across models, and most prompt variations produce few distinctive responses relative to the base prompt.The “iterate” variation has the strongest positive impact, suggesting that explicitly using prior generations can find more different responses.
- Quality vs. factuality: Creative utility trades off with factuality: GPT-5 has lower utility than Claude-Sonnet-4.6 but higher factuality.The analysis varies a factuality threshold and computes factuality-adjusted utility while treating factuality separately from the quality measure.
- Quality vs. factuality: As the factuality cutoff increases, creative utility consistently falls; at the strictest cutoff, GPT-5.5 and Sonnet-4.6 perform best while open-source models drop substantially.The steepest decline occurs from thresholds 0.5 to 0.7 because many paths contain only two or three relations.
- Search behavior: Reasoning traces show enumeration, cross-domain connections, and template-based search alongside repeated entities and paths, indicating inefficient token use.The trace analysis identifies a gap in how models search conceptual spaces for connections.
K Tests for human creativity and diversity
The paper argues that standard human creativity tests are saturated for frontier language models. It evaluates RAT, AUT, and DAT using task-specific accuracy or originality measures and reports high performance on all three.
- Remote Associations Test: RAT presents three seemingly unrelated stimulus words and asks for a fourth word related to all three.
- Alternate Uses Task: AUT asks subjects to list as many unique uses as possible for a common object, measuring originality here.The broader task evaluates fluency, originality, flexibility, and elaboration, but this work focuses only on originality.
- Diversity Association Task: DAT asks subjects to name ten nouns that differ as much as possible from one another.
- Evaluation: RAT is scored by accuracy, whereas AUT and DAT use originality measures based on embedding cosine distance.For AUT, originality measures novelty relative to the prompt; for DAT, it measures pairwise difference among response words.
- Results: GPT-4.1-mini and GPT-5-mini achieve very high RAT accuracy and high AUT and DAT originality.