Source-linked AI summary
CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges
Zi-Han Wang, Lam Nguyen, Zhengyang Zhao, Mengyue Yang, Chengwei Qin, Yujiu Yang, Linyi Yang
TL;DR
High-quality pre-training data is saturating, while evolutionary systems lack rigorous quantitative evaluation of creativity. The paper introduces CreativeBench, an automated code-generation benchmark grounded in Boden’s framework, and finds distinct scaling and reasoning patterns before proposing EvoRePE for inference-time steering.
Problem
Evolutionary systems lack rigorous measurement that objectively distinguishes creativity from hallucination and provides grounded quantitative evaluation.
Method
CreativeBench evaluates combinatorial and exploratory code-generation creativity using automated reverse engineering and self-play, executable verification, and a Quality-times-Novelty score.
Results
Scaling favors combinatorial creativity over exploration, larger models become more correct but less divergent, and reasoning primarily benefits exploratory creativity.
Takeaways & Limitations
EvoRePE provides a plug-and-play way to steer models toward more creative behaviors by internalizing evolutionary search patterns.
Takeaways & Limitations
CreativeBench has limitations including its language scope, training scope, and potential generator bias.
Abstract
from arXiv · showhide
The saturation of high-quality pre-training data has shifted research focus toward evolutionary systems capable of continuously generating novel artifacts, leading to the success of AlphaEvolve. However, the progress of such systems is hindered by the lack of rigorous, quantitative evaluation. To tackle this challenge, we introduce CreativeBench, a benchmark for evaluating machine creativity in code generation, grounded in a classical cognitive framework. Comprising two subsets -- CreativeBench-Combo and CreativeBench-Explore -- the benchmark targets combinatorial and exploratory creativity through an automated pipeline utilizing reverse engineering and self-play. By leveraging executable code, CreativeBench objectively distinguishes creativity from hallucination via a unified metric defined as the product of quality and novelty. Our analysis of state-of-the-art models reveals distinct behaviors: (1) scaling significantly improves combinatorial creativity but yields diminishing returns for exploration; (2) larger models exhibit ``convergence-by-scaling,'' becoming more correct but less divergent; and (3) reasoning capabilities primarily benefit constrained exploration rather than combination. Finally, we propose EvoRePE, a plug-and-play inference-time steering strategy that internalizes evolutionary search patterns to consistently enhance machine creativity.
1 Introduction
CreativeBench addresses the lack of rigorous creativity measurement by evaluating combinatorial and exploratory creativity in code generation through executable, quantitatively grounded tasks. Its analysis finds distinct effects of scaling and reasoning, and introduces EvoRePE for inference-time creativity steering.
- Existing evaluations often conflate creativity with hallucination, use insufficiently complex tasks, and lack grounded quantitative metrics for evolving systems.
- CreativeBench targets combinatorial creativity, which recombines familiar concepts, and exploratory creativity, which discovers possibilities within a structured conceptual space.
- The benchmark uses executable code, an automated construction pipeline, and a creativity score defined as the product of Quality and Novelty.Quality is verified through sandboxed execution and LLM judging, while novelty is measured against appropriate baselines.
- 89.1% instance validity and Spearman’s ρ = 0.78 indicate strong data validity and agreement between automated and human creativity rankings.
- Scaling substantially improves combinatorial creativity but provides limited gains for exploratory creativity, while reasoning primarily benefits constraint-driven exploration.
- EvoRePE is a plug-and-play inference-time method that extracts a creativity vector from evolutionary trajectories to steer latent representations.Its creativity gains are reported as orthogonal to the underlying evolutionary strategy.
2 Related Work
Prior work largely evaluates code-generation systems through correctness and has only partially addressed creative behavior. CreativeBench combines creativity-focused evaluation with automated construction and representation steering to address these gaps.
- Code-generation benchmarks predominantly judge functional correctness with Pass@k, often overlooking creative dimensions of problem solving.
- Existing creativity evaluations use divergent and convergent thinking but still face insufficient task complexity and unreliable quantitative measurement for evolutionary systems.
- Evolutionary approaches pair language models with programmatic evaluators or evolutionary algorithms to search for novel, learnable artifacts.
- Representation engineering commonly steers alignment-related behaviors, while its use for enhancing creativity alongside evolutionary algorithms remains unclear.
- CreativeBench addresses these gaps with a creativity benchmark, and the paper reports that a latent creativity direction can steer models toward creative solutions.
3 CreativeBench
CreativeBench is constructed from seed code tasks through reverse engineering, solution verification, self-play constraint generation, and multi-stage quality filtering. The resulting benchmark covers diverse programming scenarios while retaining substantial difficulty and undergoing manual validation.
- CreativeBench starts from 196 Python problems and spans 14 programming domains using a scalable reverse engineering and self-play construction pipeline.
- CreativeBench-Combo: For combinatorial tasks, the pipeline fuses code components from different domains, executes candidate solutions in a sandbox, and derives high-difficulty problem descriptions from verified composite code.
- CreativeBench-Combo: Test cases are generated from verified solutions and converted into assertion statements containing sandbox-derived ground-truth outputs.
- CreativeBench-Explore: For exploratory tasks, a Constraint Generator progressively adds negative constraints while a Solver refines solutions, advancing only when the constrained instance remains valid.
- Quality control: A three-stage filter removes overly easy problems, audits specification and test quality, and detects semantic redundancy through embedding similarity.
- Manual verification: Expert review found an 89.1% data validity rate, while the authors acknowledge that automated LLM-based quality control cannot guarantee 100% accuracy.
4 Experimental Setup
The experiments evaluate foundation models and evolutionary baselines using a unified creativity score that rewards both quality and novelty. CreativeBench measures combinatorial and exploratory novelty, while EvoRePE extracts evolutionary shifts into an inference-time steering vector.
- Models and Baselines: The study evaluates foundation models, zero-shot prompting, AlphaEvolve, and GEPA as baselines.AlphaEvolve uses island-style genetic search with iterative mutation and selection while maintaining population diversity.
- Evaluation Metrics: Creativity is defined as the expected product of solution quality and novelty across samples.The multiplicative score assigns low values to solutions that are correct but routine or novel but incorrect.
- Evaluation Metrics: Quality is measured with sandbox-validated Pass@1, a lower-bound estimate of single-sample success probability.All generated solutions are executed and validated inside a sandbox.
- Evaluation Metrics: Novelty measures deviation from observed or baseline solutions using CodeXEmbed distance complemented by character-level 4-gram distance.The benchmark separately measures exploratory deviation from an unconstrained baseline and combinatorial deviation from source solutions.
- Evaluation Metrics: Manual validation finds high agreement between automated and expert creativity rankings, with Spearman’s ρ = 0.78.The comparison was conducted on a sampled subset.
- EvoRePE: EvoRePE extracts a principal-component creativity vector from activation shifts between standard and evolutionarily optimized prompts, then steers the residual stream at inference time.The method is training-free and uses a steering strength α to control the intervention.
5 Results
CreativeBench remains difficult for state-of-the-art models and reveals different scaling effects across combinatorial and exploratory creativity. EvoRePE also improves creativity scores independently of the evolutionary strategy, while evolutionary search remains computationally costly and foundation-model-limited.
- CreativeBench Results: Pass@1 remains below 60% for Gemini-3-Pro on both CreativeBench subsets, indicating substantial benchmark difficulty.The benchmark uses high-difficulty seeds to reduce memorization and elicit creative problem solving.
- Scaling Effects: Scaling substantially improves combinatorial creativity but provides limited gains for exploratory creativity.The reported asymmetry is associated with richer recombination benefiting from expanded model capacity, while exploration requires escaping dominant solution patterns.
- Scaling Effects: Larger models become more correct but less divergent, a pattern described as convergence-by-scaling.This result links stronger model priors with more stable, routine solutions.
- Limitations and Implications: Evolutionary algorithms face computational overhead from novelty search and remain constrained by the creativity of the underlying foundation model.EvoRePE addresses these bottlenecks through training-free inference-time injection of a creativity vector.
- EvoRePE Results: EvoRePE raises Qwen2.5-7B-Instruct’s CreativeBench-Combo creativity score from 0.174 to 0.192 with AlphaEvolve.The method also provides consistent gains with vanilla prompting, without evolutionary search.
6 Discussion and Analysis
Scaling improves correctness more reliably than divergence, while reasoning helps exploratory creativity more than combinatorial creativity. The analysis frames foundation models as evolutionary operators evaluated through joint quality-and-novelty fitness.
- Scaling and convergence: Scaling raises Pass@1 while Novelty declines or plateaus, producing “Convergence-by-Scaling.”Larger models become more correct and standardized as their generations concentrate toward high-probability modes.
- Scaling and convergence: Novelty and functional quality behave as largely orthogonal dimensions: scaling strengthens correctness without systematically increasing departure from training priors.Smaller models can yield higher novelty through higher-variance trajectories, typically at the cost of correctness.
- Reasoning effects: Reasoning provides almost no benefit on combinatorial creativity but significantly improves exploratory creativity under constraints.The analysis attributes this contrast to retrieval and composition for cross-domain fusion versus structured search for constrained alternatives.
- Evolutionary interpretation: Foundation models act as mutation operators, while exploratory creativity corresponds to mutation and combinatorial creativity to recombination.The framework computes fitness as a joint score of quality and novelty.
- Future directions: Extending evaluation beyond executable code requires domain-appropriate representations and robust criteria for both quality and novelty.The paper identifies storytelling, music, visual design, 3D artifacts, games, and scientific discovery as future domains.
7 Conclusion
CreativeBench evaluates combinatorial and exploratory creativity in evolutionary code-generation systems and exposes distinct trade-offs in modern foundation models. The paper identifies convergence-by-scaling and reasoning benefits concentrated in exploration, then proposes EvoRePE to steer models toward more creative behavior.
- Conclusion: CreativeBench evaluates combinatorial and exploratory creativity in evolutionary systems using Boden’s cognitive framework.The benchmark is presented as the paper’s framework for analyzing machine creativity.
- Conclusion: Increasing model scale improves functional correctness but suppresses divergence, an effect termed Convergence-by-Scaling.The conclusion presents this as a central trade-off identified by the benchmark analysis.
- Conclusion: Advanced reasoning capabilities primarily benefit exploratory rather than combinatorial creativity.The reported distinction separates constraint-driven exploration from cross-domain combination.
- Conclusion: EvoRePE steers models toward more creative behaviors by internalizing evolutionary search patterns.It is described as a plug-and-play representation engineering strategy.
Limitations
The paper’s scope is limited by its Python-focused benchmark, evaluation-only setting, and possible generator bias. Its creativity framing also emphasizes P-Creativity and automated code-based assessment, with broader domains requiring additional evaluation machinery.
- Scope: CreativeBench is currently instantiated in Python, although the pipeline is described as extensible to other languages and paradigms.Python is selected for concise syntax and mature tooling supporting controlled novelty and correctness analysis.
- Scope: The study evaluates self-evolving code-generation systems rather than training models with CreativeBench.Limited computational resources prevented large-scale training or fine-tuning experiments.
- Bias: Because CreativeBench is automatically constructed, it may inherit generator bias from the underlying LLM-based pipeline.The paper notes that such bias can be measured and mitigated and may be less likely to overturn ranking comparisons under appropriate conditions.
- Creativity scope: CreativeBench focuses on P-Creativity rather than H-Creativity, measuring novelty relative to the model’s own priors or source components.The benchmark defines novelty through distance from a standard baseline representing default or high-probability behavior.
D.1.3 Human Evaluation
The human evaluation and robustness checks assess whether CreativeBench’s automated judgments align with expert assessment and whether novelty signals resist superficial code changes. Results support ranking consistency and embedding-signal stability, while noting residual sensitivity concerns.
- Human evaluation: 89.1% overall validity and Spearman’s ρ = 0.78 indicate high consistency between automated creativity rankings and expert judgments.The human-labeled study used three computer-science master’s students and daily author checks.
- Paired-seed evaluation: EVOREPE yields significant gains for Combo and Explore Pass@1 and Creativity across paired-seed tests.The tests use N = 10 matched random seeds and report two-sided paired t-tests with df = 9.
- Caveat: The character-level 4-gram novelty term may remain sensitive to identifier renaming, formatting, comments, and length-related scale artifacts.Additional stress tests are provided to characterize these effects under non-semantic perturbations and length variation.
- Novelty metric: The novelty score combines bounded semantic embedding distance with bounded character 4-gram Jaccard distance.The embedding term is cosine distance, while the n-gram term lies in [0, 1].
- Robustness checks: Canonicalizing whitespace, indentation, and comment-only changes reduces sensitivity to superficial formatting rewrites.The preprocessing preserves sensitivity to genuine lexical rewrites.
- Robustness checks: Under typical non-semantic edits and moderate length variation, the embedding-based novelty signal remains stable while cross-model distances are substantially larger.The comparison uses 120 shared combination problems across GPT-4.1-nano, Gemini-2.5-pro, and Qwen2.5-Coder-1.5B.
F.2 Role and robustness of the character-level 4-gram term
The character-level 4-gram term complements semantic embeddings by capturing lexical novelty, while robustness analyses bound its sensitivity to superficial edits and length mismatch. Normalization leaves model-level creativity conclusions essentially unchanged.
- The hybrid novelty metric combines semantic deviation from embeddings with lexical surface-level variation from the char-4gram term.The analyses specifically test sensitivity to superficial edits and length mismatch.
- Pearson r = 0.1168 for dngram versus length ratio, and Pearson r = 0.1052 for dngram versus absolute length difference across 54,940 source–combo pairs.
- After canonicalization, typical non-semantic edits change dngram only slightly, with more noticeable increases under substantial comment-only length growth.
- Aggressive whitespace-only reformatting is treated as an adversarial sensitivity test for the lexical metric.
- The Spearman correlation between original and normalized creativity rankings is 0.9989, with a maximum rank shift of 1.
- Overall, sensitivity to superficial edits is bounded, length mismatch correlates weakly with dngram, and model-level conclusions remain unchanged after normalization.
G.1.1 Task Specification and Constraints
CreativeBench uses constrained, executable code-generation tasks to test whether models can restructure algorithms rather than merely rewrite syntax. The case studies show alternative search procedures and substantial mathematical, syntactic, and semantic restructuring under blocked idioms.
- Problem 126: Forbidding binary search in the Maximum Hamburgers task forces alternative search procedures while preserving the monotone feasibility structure.
- Problem 126: The constrained hamburger solution computes a safe upper bound and uses coarse-to-fine step-down refinement instead of midpoint selection and repeated halving.Large steps correct order-of-magnitude errors, while smaller steps finalize the maximum feasible output.
- Problem 126: The constrained hamburger solution exhibits algorithmic restructuring through analytic bounding and a step-down search for the maximum feasible output.
- Problem 192: Qwen2.5-72B-Instruct achieves a Creativity Gap of 0.3999 on the Temperature Conversion Table task under scorched-earth constraints.
- Problem 192: The model decomposes the forbidden 5/9 fraction analytically and uses decimal arithmetic to preserve precision under the constraint.
- Problem 192: Banning loops induces recursive traversal, while banning high-level formatting leads to primitive stream and manual alignment operations.
- Problem 192: The Temperature Conversion case reports high novelty of 0.40 as structural divergence from the baseline, illustrating creativity beyond instruction inertia.
- Evaluation pipeline: The benchmark evaluates sandboxed candidate modules with assertion-based tests and audits specification, signature, objective, and requirement alignment.
I.2 Robustness to Injection Layer
EvoRePE remains effective across a contiguous mid-to-late layer band rather than depending on one injection point, with performance degrading at overly early or late layers. The supplied passages also describe a broad steering-strength robustness window.
- The layer sweep injects the same creativity vector across early, middle, and late transformer blocks while holding steering strength and decoding settings fixed.
- On a robustness split of N = 100, QWEN2.5-7B-INSTRUCT improves Creativity from 0.174 to 0.192 when injected at Layer 26.
- Layers 22–28 form a contiguous positive-gain band, with nearby examples of 0.198 at Layer 24 and 0.195 at Layer 28.
- Performance drops when intervention occurs before Layer 12 or after Layer 32, where correctness can suffer or creativity gains can diminish.
- The steering-strength sweep identifies α ∈ [0.05, 0.45] as a stable window with consistent creativity gains without sacrificing correctness.
- At α = 0.5, Pass@1 begins to decrease, suggesting over-steering.
- The paper recommends selecting layers from the stable region and α from the robustness interval using a lightweight validation sweep.