PaperlayerView exact revision on arXiv

Source-linked AI summary

Combinatorial Creativity: A New Frontier in Generalization Abilities

Samuel Schapiro, Sumuk Shashidhar, Alexi Gladstone, Jonah Black, Royce Moon, Dilek Hakkani-Tur, Lav R. Varshney

arXiv:2509.21043v5cs.AIcs.LG

TL;DR

AI systems are increasingly used for creative tasks such as scientific idea generation, but existing conceptual frameworks do not adequately address this form of generalization. The paper introduces a framework and algorithmic task for evaluating combinatorial creativity, then finds scaling and architectural patterns alongside a novelty–utility tradeoff. Its findings offer a starting point for study, while remaining limited to synthetic data and models up to 100M parameters.

  • Problem

    AI is increasingly used for scientific idea generation, but these tasks lack strong mathematical and conceptual foundations, limiting diagnosis and improvement.

    Source: §1 · p. 2

  • Method

    The paper formalizes combinatorial creativity as an open-ended task on a synthetic conceptual graph and studies decoder-only Transformers across sizes, depths, widths, and compute budgets.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Creativity improves predictably with model size and compute, fixed-budget depth and width have an optimal tradeoff, and novelty declines as utility constraints increase across studied scales.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The framework and findings provide a starting point for studying and improving AI creativity and extending the work to larger scales and more semantically grounded spaces.

    Source: §1 · p. 2

    The study focuses only on combinatorial creativity, uses synthetic data, and evaluates models only up to 100M parameters.

    Source: §6 · p. 11

Abstract

from arXiv · show

Artificial intelligence (AI) systems, and Large Language Models (LLMs) in particular, are increasingly employed for creative tasks like scientific idea generation, constituting a form of generalization from training data unaddressed by existing conceptual frameworks. Despite its similarities to compositional generalization (CG), combinatorial creativity (CC) is an open-ended ability. Instead of evaluating for accuracy or correctness against fixed targets, which would contradict the open-ended nature of CC, we propose a theoretical framework and algorithmic task for evaluating outputs by their degrees of novelty and utility. From here, we make several important empirical contributions: (1) We obtain the first insights into the scaling behavior of creativity for LLMs. (2) We discover that, for fixed compute budgets, there exist optimal model depths and widths for creative ability. (3) We find that the ideation-execution gap, whereby LLMs excel at generating novel scientific ideas but struggle to ensure their practical feasibility, may be explained by a more fundamental novelty-utility tradeoff characteristic of creativity algorithms in general. Though our findings persist up to the 100M scale, frontier models today are well into the billions of parameters. Therefore, our conceptual framework and empirical findings can best serve as a starting point for understanding and improving the creativity of frontier-size models today, as we begin to bridge the gap between human and machine intelligence.

1 INTRODUCTION

The paper frames combinatorial creativity as an important but insufficiently grounded capability for AI, especially in scientific ideation. It introduces a controlled evaluation framework and studies how model scale and architecture relate to creative performance.

1 INTRODUCTION

  • The paper situates its approach in accounts of creativity as a combinatorial process in the space of mental representations.It recalls Einstein’s view of combinatory play as central to productive thought and notes historical accounts of combining familiar concepts into novel ideas.Source: §1 · pp. 1–2
  • LLM-generated scientific ideas can be practically infeasible, make unrealistic assumptions, or omit baselines, while foundations for understanding these abilities remain weak.The authors argue this limits the ability to diagnose and improve LLM outcomes on such tasks.Source: §1 · p. 2
  • A formal framework and open-ended algorithmic task evaluate combinatorial creativity, with decoder-only Transformers studied across 1M–100M parameters and training compute budgets.The study varies model size, depth, and width in a synthetic conceptual graph testbed.Source: §1 · p. 2
  • The study finds predictable creativity gains with model size and training compute, a fixed-budget depth–width sweet spot, and a persistent novelty–utility tradeoff across studied scales.As task complexity rises, models more often violate utility constraints than produce trivially non-novel outputs.Source: §1 · p. 2

2 BACKGROUND

The background places combinatorial creativity among broader accounts of creativity and distinguishes it from classical compositional generalization. Its defining evaluation difference is open-ended assessment by degrees of novelty and utility.

2.1 BACKGROUND ON CREATIVITY

  • Creativity is framed as producing novel, useful, and surprising artifacts; Boden distinguishes combinatorial, exploratory, and transformational forms.Combinatorial creativity combines familiar concepts in unfamiliar ways, while transformational creativity restructures a conceptual space’s rules.Source: §2.1 · p. 2
  • Combinatorial creativity has roots in accounts of ideas combining or distant associations linking, and Boden later explicitly defined the term.The literature associates distant associations with more creative artifacts and models many impactful discoveries as combinatorial.Source: §2.1 · p. 3

2.2 DISTINGUISHING COMBINATORIAL CREATIVITY FROM CLASSICAL FORMS OF GENERALIZATION

  • Unlike systematicity and productivity, combinatorial creativity is open-ended and evaluates artifacts by degrees of novelty and utility rather than fixed correctness.It combines familiar components into new forms and meanings; classical CG evaluations may be closed-ended and do not measure these degrees for open-ended artifacts.Source: Table 1 · p. 4, §2.2 · pp. 4–5

3 A THEORETICAL FRAMEWORK AND OPEN-ENDED, ALGORITHMIC TASK FOR COMBINATORIAL CREATIVITY

The framework evaluates open-ended combinatorial creativity by prompting models to connect concepts in a graph while satisfying logical constraints, then scoring artifacts by novelty and utility. It also extends prior tasks through variable-length artifacts and graded measures of novelty and utility.

3 A THEORETICAL FRAMEWORK AND OPEN-ENDED, ALGORITHMIC TASK FOR COMBINATORIAL CREATIVITY

  • Models must compose labeled paths between concepts while obeying inclusion and exclusion constraints; any satisfying path is valid for evaluation.The framework represents conceptual spaces as labeled graphs and creative artifacts as walks between a prompt’s start and end concepts.Source: §3 · p. 5, §3.1 · p. 5

3.2 QUANTIFYING DEGREES OF NOVELTY

  • Novelty combines path length and label surprise, using graph-walk distance and label probabilities as controllable proxies for semantic distance.The paper adopts graph distances and label surprise to keep novelty measurement controllable.Source: §3.2 · pp. 5–6

3.3 QUANTIFYING DEGREES OF UTILITY

  • Utility rewards paths that connect the requested endpoints, include required labels, and exclude forbidden labels, with constraint counts scaling the score.The evaluation set increases difficulty through path length and constraint count, while constrained breadth-first search ensures a valid solution exists.Source: §3.3 · pp. 6–7

3.4 MEASURING CREATIVITY

  • Creativity is measured as the expected product of novelty and utility over the prompt distribution.This multiplicative score requires an artifact to be both novel and useful to score highly.Source: §3.4 · p. 7

3.5 DETAILED COMPARISON WITH SIBLING AND TRIANGLE DISCOVERY

  • Against Nagarajan et al. (2025), the framework permits variable-length artifacts and graded novelty and utility rather than fixed forms and binary novelty or coherence.Its utility constraints include labels that paths must include and labels they must exclude.Source: §3.5 · p. 7

4 EXPERIMENTS

The experiments sweep Transformer size and architecture to examine creativity across model scales. Creativity varies non-monotonically with depth and width, while greater utility constraints accompany lower novelty across the tested scales.

4.1 MODEL ARCHITECTURE

  • At 100M parameters, creativity peaks around 8 layers; width performs best at an E/L ratio between 200 and 300.Across 1M, 10M, and 100M budgets, both depth and width show non-monotonic relationships with creativity.Source: Figure 3 · p. 8
  • The study sweeps GPT-2 decoder-only Transformers at approximately 1M, 10M, and 100M parameters, varying depth, width, and attention heads.Depth and embedding width are traded off while keeping their product roughly constant; the number of heads is also varied for each configuration.Source: §4.1 · pp. 8–9
  • Across 1M, 10M, and 100M models, normalized novelty trends downward as the number of utility constraints increases.Novelty is normalized by the mean novelty of simple, single-hop paths at each constraint level.Source: Figure 4 · p. 9

5 RESULTS AND DISCUSSION

The results identify an optimal depth–width balance for creativity and a persistent novelty–utility tradeoff. Error analysis and constraint mappings connect that tradeoff to practical feasibility issues in scientific idea generation.

The existence of optimal depths and widths for creativity

  • Across all three model sizes, creativity peaks at an optimal depth; for 100M models this occurs around 8 layers, with an optimal E/L ratio of 200–300.Models with 2–4 or 12+ layers perform substantially worse for their parameter count.Source: §5 · p. 9

The novelty-utility tradeoff

  • Across all three scales, novelty declines as the number of utility constraints increases, and this tradeoff does not improve up to 100M parameters.The authors call for future study of this tradeoff at billion-parameter scale.Source: §5 · p. 9

Isolation of errors

  • At 1M and 10M, hallucinations dominate utility errors; at 100M, they fall sharply and invalid-path errors become nearly as frequent.The authors report that larger models’ utility errors become subtler and more semantic, despite fewer superficial errors.Source: Figure 5 · p. 10, §5 · p. 10

Understanding the ideation-execution gap for LLM-generated ideas

  • The novelty–utility tradeoff offers a potential explanation for why LLMs generate novel research ideas yet struggle to ensure practical feasibility.The table maps exclusion constraints to unrealistic assumptions and expensive plans, and inclusion constraints to baselines and sufficiently detailed implementation plans.Source: §5 · p. 10, Table 2 · p. 10

6 LIMITATIONS AND CONCLUSION

The paper concludes that its framework and findings offer a starting point for studying and improving AI creativity, while noting limits in creativity type, data, and model scale.

6 LIMITATIONS AND CONCLUSION

  • The study covers combinatorial creativity only, uses synthetic data that may not represent real-world creative domains, and evaluates models only up to 100M parameters.The authors identify more compute and real-world data as routes for future study.Source: §6 · p. 11
  • The authors present the framework and empirical findings as a pathway for understanding and improving creativity in modern AI models.Source: §6 · p. 11

A.1 OPEN-ENDED ALGORITHMIC TASKS

The appendix situates the framework among open-ended algorithmic evaluations and prior work on mechanisms of LLM creativity. It distinguishes this framework by evaluating degrees of novelty and utility for structurally novel artifacts.

A.1 OPEN-ENDED ALGORITHMIC TASKS

  • Unlike pathfinding tasks focused on inference, this framework evaluates degrees of novelty and utility and permits structurally novel artifacts.It extends the open-ended algorithmic task suite proposed by Nagarajan et al. (2025).Source: §A.1 · p. 14

A.2 MECHANISTIC UNDERSTANDING OF CREATIVITY IN LLMS

  • Prior studies report a weak positive correlation between sampling temperature and narrative novelty and a negative correlation with coherence; prompting methods did not significantly improve creativity, while multi-token prediction increased it.The temperature findings were reported by Peeperkorn et al. (2024), prompting results by Morain and Ventura (2025), and the pre-training objective result by Nagarajan et al. (2025).Source: §A.2 · p. 14

B.1 DATASET CONSTRUCTION

The dataset uses a synthetic labeled graph as a conceptual space and generates constrained pathfinding query–solution pairs for training and evaluation. Its construction combines edge coverage with randomized exploration and holds evaluation node pairs out of training.

B.1 DATASET CONSTRUCTION

  • The synthetic conceptual space has 17,576 three-letter nodes and 52,728 labeled edges, with average node degree approximately six.Edge labels are selected from the 26 lowercase English letters.Source: §B.1 · p. 15
  • Each dataset example pairs a query specifying start and end nodes plus inclusion and exclusion label sets with a valid labeled path.A valid path connects the specified endpoints, uses all included labels, and avoids excluded labels.Source: §B.1 · p. 15
  • Training data combines two stages: examples covering every graph edge in both directions, and randomized constrained-path queries.The randomized stage samples node pairs and constraint sets, then uses constrained BFS to find a path up to a maximum length.Source: §B.1 · p. 15
  • Evaluation node pairs are excluded from training, preventing any pair in the evaluation set from appearing in the training set.Source: §B.1 · p. 15

B.2 TRAINING AND TOKENIZATION

The models are trained as GPT-2 causal language models to generate graph paths from queries, using a graph-specific tokenizer. Training uses fixed 16-epoch runs, and evaluation generates one path per problem with greedy decoding.

Hyperparameter Choice

  • Models in the 1M, 10M, and 100M buckets receive scale-dependent learning rates, with AdamW, cosine decay, warmup, and 16 training epochs.The schedule is intended to keep training near-optimal across model sizes and facilitate fair comparisons.Source: §B.2 · p. 15

Pre-Training and Tokenization

  • Models use a standard GPT-2 architecture to conditionally generate a path P from a query Q, trained from scratch with cross-entropy on path tokens only.Query tokens are masked from the loss, so they condition generation without supervision for query generation.Source: §B.2 · pp. 15–16
  • The custom tokenizer assigns atomic graph nodes, lowercase edge labels, and syntax/control characters to separate token types.Treating concepts as indivisible units aligns tokenization with the paper’s view of creativity as recombination of known concepts.Source: §B.2 · p. 16

Evaluation

  • At each epoch’s end, greedy decoding generates one path for every problem in the structured evaluation set.Source: §B.2 · p. 16

C BROADER IMPACT AND FUTURE WORK

The authors identify unmeasured diversity and the study’s focus on combinatorial creativity as limits, and propose extending evaluation and studying other creativity forms.

Evaluating Diversity

Scaling Behavior for Exploratory and Transformational Creativity

C.1 AVENUES FOR IMPROVING MODEL CREATIVITY

The authors point to training objectives, inference-time refinement, and alternative architectures as directions for improving creative capabilities.

Pre-Training Objective

  • The effect of pre-training objectives on combinatorial creativity remains open, with prior work comparing NTP, MTP, and TOP.Nagarajan et al. (2025) found increased algorithmic creativity with MTP on two minimal tasks, while Zuhri et al. (2025) report improved scaling behavior for TOP over NTP and MTP.Source: §C.1 · p. 16

Democratizing Creative AI Through Inference-Time Techniques

  • Shashidhar et al. (2023) report up to 25.39% improvement on high-creativity open-ended tasks through iterative self-critique.The authors identify self-refinement as a promising inference-time strategy for resource-constrained settings because it requires no additional training.Source: §C.1 · p. 16

Architectural Innovations

  • Energy-Based Transformers could offer promising creativity capabilities, given their explored role in System-2 thinking and generalization.The authors connect this possibility to Energy-Based Models’ demonstrated compositional generalization abilities.Source: §C.1 · p. 16
Loading 2509.21043v5…