PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating LLM Creativity through Long-Tail Performance

Yichen Wang, Tenghao Huang, Vishakh Padmakumar, Mina Lee

arXiv:2610.12998v1cs.CL

TL;DR

LLM creativity benchmarks vary widely in task and metric design, leaving unclear whether they capture a common notion of creativity. The paper proposes long-tail performance under a reference input-output distribution as a shared target and evaluates it across benchmarks. Creativity benchmarks lie further in the tail than standard benchmarks, and higher-tail subsets align more closely with human creativity rankings.

  • Problem

    LLM creativity benchmarks differ in task and metric design, leaving uncertain whether they capture a common notion of creativity.

    Source: §1 · p. 2

  • Method

    The paper formalizes a shared target as task performance in the long tail of a reference input-output distribution and operationalizes tailness using n-gram novelty.

    Source: §1 · p. 2, §3.1 · p. 3, §4 · p. 4

  • Results

    Creativity benchmarks lie further in the tail than standard benchmarks, and higher-tail subsets align more closely with the LM Arena creative-writing ranking.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    Most creativity-promoting methods increase output novelty without consistently improving tail quality, indicating room to improve creativity through stronger tail performance.

    Source: §1 · p. 2

    q@Tail is a diagnostic approximation rather than a replacement for creativity benchmarks, and its signal cannot be attributed exclusively to creativity.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common notion of creativity. We propose that this shared target can be formalized as task-specific performance in the long tail of a reference input-output distribution, which describes how common input-output pairs are relative to a chosen source of text. We ground this proposition in the novelty-quality product formulation and evaluate it empirically across 15 benchmarks and 26 open-weight models, using n-gram absence from the pretraining data as a novelty proxy. Our results show that creativity benchmarks lie further in the tail regions than standard benchmarks. Moreover, on both creativity and standard benchmarks, model rankings from higher-tail subsets of each benchmark align more closely with human creativity judgments in LM-Arena. Building on these findings, we introduce q@Tail($α$), a lower-cost creativity diagnostic that measures mean task quality on the tail fraction of an existing benchmark. Applying it to creativity-promoting methods reveals substantial room to improve LLM creativity through stronger tail performance.

1 Introduction

LLM creativity benchmarks vary in task and metric design, leaving unclear whether they capture a common notion of creativity. This section proposes long-tail performance as that shared target and outlines empirical support and a lower-cost diagnostic.

1 Introduction

  • The lack of consensus on human creativity definitions and the variation across LLM task and metric designs motivate seeking a shared evaluation target.A common target could provide a principled basis for designing, validating, and comparing evaluations.Source: §1 · pp. 1–2
  • Creativity benchmarks shift evaluation toward low-probability regions of a reference input-output distribution, unlike standard benchmarks that focus on common inputs and outputs.Tail regions include improbable inputs, improbable outputs given their inputs, or otherwise improbable input-output pairs under the reference distribution.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper proposes long-tail task performance under a reference input-output distribution as a shared target across heterogeneous creativity benchmarks.The reference distribution describes how common input-output pairs are relative to a chosen text source.Source: §1 · p. 2
  • Higher-tail benchmark subsets align more closely with LM Arena’s creative-writing ranking, while most creativity-promoting methods do not consistently improve tail quality.The paper identifies stronger tail performance as an area for improving LLM creativity.Source: §1 · p. 2
  • The authors survey LLM creativity evaluations and characterize their design variation.They also provide theoretical and empirical support for the proposed long-tail target.Source: §1 · p. 2
  • The paper introduces a lower-cost tail-performance diagnostic for existing benchmarks and uses it to assess creativity-promoting methods.The diagnostic measures quality on a benchmark’s tail fraction.Source: §1 · p. 2

2 What is Missing in Literature?

Prior work documents substantial variation in LLM creativity tasks and metrics, with studies sometimes reaching contrasting conclusions. This section identifies the unresolved question of whether those evaluations share a common target.

2 What is Missing in Literature?

  • LLM creativity evaluations vary substantially in task and metric design and can produce contrasting conclusions across studies.The review organizes these choices into a nine-dimensional taxonomy that highlights potential sources of disagreement.Source: §2 · p. 2
  • Although novelty and quality operationalize creativity on the metric side, the implicit target of tasks designed to elicit creativity remains unexplained.The paper proposes long-tail performance under a reference input-output distribution as a common task-level target.Source: §2 · p. 3

3 Evaluate LLM Creativity through Long-Tail Performance

The paper defines novelty relative to a chosen reference distribution over input-output pairs, then connects benchmark creativity to quality across regions of that distribution. Because higher-novelty regions receive more weight, quality differences there contribute more to overall creativity.

3.1 Definition

  • A reference distribution specifies how common input-output pairs are relative to a chosen text source, and factorizes into input and conditional-output probabilities.This gives each pair a position in a two-axis space with peak, input-tail, output-tail, and joint-tail regions.Source: §3.1 · p. 3
  • The prompt-level novelty score aggregates pair novelty across evaluated outputs, summarizing how far each prompt and its outputs lie in the reference tail.The aggregation is fixed across benchmarks and models and maps scores to [0, 1].Source: §3.1 · p. 3

3.2 Theoretical connection

  • Under the novelty-quality product formulation, overall creativity aggregates model quality across nested novelty regions, with higher-novelty quality differences contributing more.A quality advantage on an input receives total weight equal to that prompt’s novelty score.Source: §3.2 · pp. 3–4
  • q@Tail measures mean quality on a selected top-α benchmark subset and approximates the relative ranking from overall creativity when selected-tail differences dominate or align with other regions.The paper verifies this condition empirically in §6.Source: §3.2 · p. 4

4 Experimental Setup

The evaluation uses corpus-based n-gram novelty to locate benchmark prompts in a reference distribution, then compares model performance and rankings across 15 benchmarks and 26 open-weight models.

Reference distribution

  • The Pile serves as the reference text source, with InfiniGram-Mini corpus-occurrence queries determining n-gram absence for novelty scoring.Because the Pile predates the evaluated benchmarks, this choice mitigates direct contamination that could inflate the measurement.Source: §4 · p. 4

Novelty metric

  • Novelty is the fraction of a text’s n-grams absent from the Pile, used as a tractable proxy for how far the text lies in the reference tail.The analysis uses n = 4; checks at n = 5 and 6 are consistent, and output novelty is averaged across evaluated models before prompt scoring.Source: §4 · p. 4

Models

  • The study evaluates 26 recent open-weight models spanning model families, sizes, and instruction- or reasoning-tuned configurations.Most selected models also appear on the LM Arena creative-writing leaderboard used as an external human-preference reference.Source: §4 · p. 4

Benchmarks

  • The evaluation covers 15 benchmarks across free-form writing, mathematics reasoning, and free-form QA, pairing creativity benchmarks with standard benchmarks in the same domains.The set includes 10 writing, 2 mathematics, and 3 QA benchmarks.Source: §4 · p. 4

Performance evaluation

  • Within each task domain, performance metrics are held fixed, and model rankings are compared with the LM Arena creative-writing ranking using Kendall’s τ.Math and QA use exact match; open-ended generation uses normalized Skywork-Reward-V2-Llama-3.1-8B reward scores, without comparing absolute quality across domains.Source: §4 · p. 5

5 Creativity Benchmarks Lie in the Tail of Pref

Across writing, mathematics, and free-form QA, creativity benchmarks occupy higher-novelty regions than standard benchmarks. The paper relates this shift to distinct elicitation operations and reports that higher-tail subsets align more closely with human rankings.

5 Creativity Benchmarks Lie in the Tail of Pref

  • Creativity benchmarks shift toward higher input and/or output novelty than standard benchmarks across all three task domains.The pattern is stable across n = 4, 5, and 6 and across model families; writing-domain distribution tests report all p < 0.001.Source: §5 · p. 5

5.1 Three elicitation operations, three tail sub-regions

  • Curation selects rare inputs, shifting benchmarks toward higher input novelty and the input-tail region.CreativeMath is an example: experts filter for problems requiring non-standard or out-of-the-box thinking.Source: §5.1 · p. 6, Figure 3 · p. 6
  • Prescription leaves inputs largely unchanged but requests creative outputs, shifting evaluation toward the output-tail region.Examples include asking for creative writing or an alternative solution.Source: §5.1 · p. 6, Figure 3 · p. 6
  • On Arena-Hard Creative Writing, tail bins 4–5 reach Kendall’s τ of +0.39 and +0.33, above peak bins 1–3 at +0.06 to +0.15.The trend generally holds across other benchmarks and choices of joint-novelty aggregator.Source: Figure 4 · p. 6
  • Constraint adds restrictions that make prompts and acceptable outputs less common, shifting evaluation toward the joint-tail region.Examples include requiring a specific setting and flashbacks or forbidding an algorithm.Source: §5.1 · p. 6, Figure 3 · p. 6

6 Tailness Contributes to More Accurate Creativity Evaluation

The section tests whether benchmark tailness predicts agreement with human creativity rankings. Higher-tail subsets align more closely within benchmarks, while the across-writing-benchmark association is positive but weaker.

6.1 Within each benchmark, tailness predicts agreement with humans

  • On Arena-Hard Creative Writing, tail bins 4–5 reach τ = +0.33 versus +0.06 to +0.15 for peak bins; the same pattern holds on GPQA-Diamond.At ten-bin resolution, novelty-bin association with human-ranking agreement is significant on both benchmarks (Spearman ρ = 0.661–0.855, p = 0.0016–0.0376).Source: Figure 5 · p. 7, §6.1 · p. 7
  • Within each benchmark, examples are binned by shared novelty score, and models’ mean-quality rankings are compared with the LM Arena creative-writing ranking.The analysis uses Kendall’s τ; the visualization divides examples into five equal-sized bins from peak to most tail.Source: §6.1 · p. 7, §6 · p. 7
  • The observed alignment motivates q@Tail(α), mean task quality on a benchmark’s top-α tail selected by the shared score, as a diagnostic using existing prompts.It does not require redesigning the benchmark or collecting new prompts.Source: §6.1 · p. 7

6.2 Across writing benchmarks, the association is positive

  • Across eight writing benchmarks, the association between benchmark tailness and agreement with human rankings is positive but weaker (Spearman ρ = 0.26).Arena-Hard Creative Writing, the most-tail benchmark, reaches τ = 0.37 (p = 0.029), while near-low-novelty Alpaca reaches τ = 0.16 and does not significantly agree (p = 0.37).Source: §6.2 · p. 7
  • Tailness is a creativity-relevant indicator rather than a complete characterization, and its alignment may not be specific to creativity or separable from general capability.Across-benchmark comparisons are weaker, and benchmark differences may affect rankings independently of tailness.Source: §6.2 · pp. 7–8

7 Diagnosing Creativity-Promoting Methods

The section uses q@Tail to assess whether creativity-promoting methods maintain quality in rare regimes, alongside novelty gains on familiar inputs. Evaluated methods show a quality–novelty trade-off, leaving room to improve tail quality.

7 Diagnosing Creativity-Promoting Methods

  • The diagnostic is applied to test-time and training-time methods across a mixture of creativity benchmarks and four model families.The methods include sampling, temperature scaling, prompting and reasoning interventions, and CrPO-SFT and -DPO.Source: §7 · p. 8
  • The evaluation distinguishes maintaining quality on tail inputs or outputs from eliciting more novel outputs on familiar, high-probability inputs.These capabilities are measured as Quality at Tail, q@Tail(α), and Novelty at Peak, respectively.Source: §7 · p. 8
  • Most evaluated methods increase Novelty at Peak but have lower Quality at Tail than the vanilla model; min-p sampling raises Quality at Tail by 17.7% while reducing Novelty Overall by 16.3% and Novelty at Peak by 44.2%.No method improves both tail quality and peak novelty in the aggregate results.Source: §7 · p. 8
  • Higher tail quality in sanity checks and with min-p sampling points to an opportunity to improve tail quality while preserving novelty.The section notes that most methods’ novelty gains often coincide with lower quality on uncommon inputs or outputs.Source: §7 · pp. 8–9

8 Creativity is not Identical to Difficulty

The section distinguishes novelty from difficulty as separate properties of benchmark inputs. Its comparisons find low overlap between novelty-selected and difficulty-selected examples.

8 Creativity is not Identical to Difficulty

  • On MATH and Cambridge CEFR, novelty and difficulty splits have low overlap (0.07 and 0.23), near random; the same pattern holds under model-based difficulty on AIME and WritingBench.Source: §8 · p. 9
  • Comparing difficulty and novelty splits on MATH and Cambridge CEFR, overlap remains low (0.07 and 0.23), near random; the same pattern holds under model-based difficulty on AIME and WritingBench.Source: §8 · p. 9
  • Difficulty analysis asks whether tail inputs challenge a fixed model, whereas this paper asks whether stronger tail performance aligns with human creativity judgments across models.The authors describe these as different axes of inquiry.Source: §8 · p. 9

9 What Can We Learn?

The section draws implications for benchmark design and model development. It recommends matching reference distributions and benchmark coverage to intended use, while identifying tail performance as a potential route to greater human-perceived creativity.

9 What Can We Learn?

  • A benchmark’s reference distribution determines what counts as novel, so its choice should reflect the intended audience or application.The paper contrasts the Pile for general users with a reference emphasizing relevant literature for scientific researchers.Source: §9 · p. 9
  • Benchmark designers can examine whether evaluations cover relevant tail regions and specify design choices to support replication and comparison.Curation, prescription, and constraint can help cover uncommon inputs, outputs, and their combinations; metric choices also trade off cost and reproducibility.Source: §9 · p. 9
  • Improving model performance in tail regions is a potential path to greater human-perceived creativity, but decoding or prompting can raise output rarity without necessarily improving rare-regime quality.The paper characterizes strengthening high-quality performance in rare regimes as more challenging and potentially more substantive.Source: §9 · p. 9

Limitations

The paper’s long-tail operationalization and empirical validation have important scope boundaries: novelty proxies and quality estimators may affect estimates, and human-alignment evidence is limited.

Limitations

  • N-gram statistics from the Pile capture surface rarity, not semantic, conceptual, or culturally grounded novelty; other reference distributions could yield different tail estimates.The selected quality estimator may also introduce architecture, length, style, and domain biases, and robustness across judges remains underexplored.Source: Limitations · p. 9
  • The strongest human-alignment analysis uses LM Arena creative-writing rankings, which may not represent creativity judgments in mathematics, science, design, or other domains.Those rankings reflect holistic preferences from self-selected users rather than judgments that isolate creativity, and benchmark-level correlations are noisy.Source: Limitations · pp. 9–10
  • The creativity-promoting-method evaluation covers only open models with 7–8B parameters, so findings may not extend to larger or frontier closed models.The authors note that methods with complex instructions, such as meta-prompting, may behave differently on those models.Source: Limitations · p. 10
  • q@Tail is a diagnostic approximation, not a replacement for carefully designed creativity benchmarks or a creativity measure disentangled from general capability.High tail performance may also reflect general capability, task-specific skills, domain knowledge, instruction following, or robustness to rare inputs.Source: Limitations · p. 10

Ethical Considerations

The material states that the study introduces no direct ethical harms, organizes creativity evaluations by task and metric, and examines whether tail performance tracks creativity-relevant behavior.

Ethical Considerations

  • The study reports no direct ethical harms and focuses on evaluating LLM creativity benchmarks and proposing a long-tail-performance diagnostic.Source: Ethical Considerations · p. 10

A Taxonomy

  • The taxonomy organizes creativity-evaluation experiments into Task and Metric dimensions to support systematic comparison of existing and new designs.Its dimensions aim to distinguish setups that may lead to divergent empirical conclusions while covering existing configurations at useful granularity.Source: A Taxonomy · p. 15, Table A1 · p. 17

A.1 Task / A.2 Metric

  • A task is defined by its objective and knowledge domain; changes in tone, theme, wording, or formatting are prompt realizations, not new tasks.A metric is an algorithmic procedure that converts one output into quantitative evaluative signals; task-level results average metrics across prompts.Source: §A.1 · p. 15, §A.2 · p. 16

METRIC

  • The taxonomy distinguishes reference sources, evaluated outputs, creativity types, granularity, and judges, including corpus-based novelty and human or model evaluation.It also distinguishes process from outcome evaluation and monolithic from compositional creativity measures.Source: METRIC · pp. 18–19

B.1 Creativity is not Identical to Difficulty

  • Novelty and difficulty subsets overlap near or below random baselines across human- and model-based difficulty measures, so the paper treats them as largely distinct.On MATH, novelty–difficulty overlap was 0.07 versus a 0.17 random baseline; on Cambridge CEFR it was 0.23 versus 0.12.Source: §B.1 · p. 20
  • The authors do not claim creativity is orthogonal to capability: splits can overlap, and tail prompts may be harder even as tail performance tracks between-model creative rankings.They distinguish difficulty on the input axis from ranking agreement on a fixed tail set.Source: §B.1 · p. 20

B.2 Creativity and General Model Capability

  • LM Arena creative-writing rankings strongly correlate with overall and capability-specific rankings, so q@Tail’s signal is not established as creativity-specific.The authors interpret q@Tail as a diagnostic approximation rather than a disentangled creativity factor.Source: §B.2 · pp. 20–21

C Models and Benchmarks / D Diagnosing creativity-promoting methods settings

  • The appendix says comparisons were standardized across diverse standard and creativity benchmarks; promoting methods were evaluated on four representative 7–8B open-weight models, with thinking enabled by default for two.CoT, a quality-oriented intervention, scored 0.539 on Quality at Tail versus 0.474 for the vanilla model; Claude Opus 4.6 High, a separate model comparison, scored 0.610.Source: C Models and Benchmarks · p. 21, D Diagnosing creativity-promoting… · p. 21

E.2.1 Robustness to the Number of Bins

At finer bin resolution, higher-tail subsets often align more closely with the LM Arena creative-writing ranking, though the pattern varies across benchmarks and reference measures.

E.2.1 Robustness to the Number of Bins

  • Arena-Hard agreement rose from approximately zero in early bins to τ = 0.27–0.49 in bins 7–10; GPQA-Diamond reached τ = 0.43 and 0.45 in bins 7 and 8.Across ten bins, joint novelty and ranking agreement had positive Spearman associations on both benchmarks (ρ = 0.661–0.855, p = 0.0016–0.0376); adjacent-bin increases were not universal.Source: §E.2.1 · pp. 21–22, Table A5 · p. 23

E.2.2 Results on Additional Benchmarks

  • AIME-All, BookMIA, and Braingle had higher agreement in their most-tail than most-peak bins, while Alpaca showed no endpoint difference and StoryGen and TinyStories stayed positively aligned across bins.Results vary across benchmarks, and tailness is not a universally monotonic predictor of agreement.Source: §E.2.2 · p. 22, Figure A2 · p. 25

E.3 Base-Model Perplexity as an Alternative Novelty Proxy

  • Base-model perplexity had a near-zero, directionally inconsistent relationship with the LM Arena creative-writing ranking, so it was not a stable alternative novelty proxy here.This does not establish that n-gram novelty is uniquely correct.Source: §E.3 · pp. 22–23

F Interpretation of q@Tail

  • q@Tail uses a shared benchmark-level novelty score to select tail examples, then measures quality there; its integral identity applies to that shared-weight score.The diagnostic reuses benchmark prompts but still requires generation, novelty computation, and quality evaluation; across eight writing benchmarks, tailness–agreement association was positive but weak and not statistically significant (Spearman ρ = 0.26, p = 0.53).Source: F Interpretation of q@Tail · p. 23, Table A7 · p. 26
Loading 2610.12998v1…