PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

Xiangyu Wang, Jin Wu, Xiaoyu Li, Chanjin Zheng, Yifeng Zhou

arXiv:2609.03432v1cs.CL

TL;DR

Automated evaluation of complex, interdependent creativity tasks remains underexplored, while existing methods either require task-specific training or judge raw responses directly. CreaEval separates response analysis from evidence-based scoring; across four LLMs, it consistently outperforms all baselines, reaching 0.64 quadratic weighted kappa (QWK) on CGPST, 0.17 above the second-best supervised-training baseline.

  • Problem

    Automated evaluation of complex, interdependent creativity tasks remains underexplored; existing methods either require task-specific training resources and annotated datasets or map raw responses directly to scores.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    CreaEval separates evaluation into memory-augmented analysis, which structures multi-step responses with cross-step memory, and evidence-based judging using that evidence alone.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Across four LLMs, CreaEval consistently outperforms all baselines, reaching 0.64 quadratic weighted kappa (QWK) on CGPST, 0.17 above the second-best supervised-training baseline.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Further analysis finds that decoupling enhances scoring stability across dimensions and mitigates verbosity and leniency biases in LLM-as-a-Judge.

    Source: §1 · p. 2

    Experiments are limited to CGPST among multi-step creativity datasets, since it is the only publicly available benchmark of that type.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose CreaEval, an automated creativity evaluator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: Memory-augmented Analysis, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and Evidence-based Judging, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments show that CreaEval achieves an average performance improvement of 22.74% over the second-best baselines across CGPST and two classic simple creativity tasks, demonstrating its generalizability. The code is available at https://github.com/Jaong/CreaEval.

1 Introduction

CreaEval addresses the difficulty of reliably evaluating complex, subjective creativity tasks by separating response analysis from evidence-based scoring. The introduction reports improved human–LLM agreement on CGPST and greater scoring stability with reduced bias.

Introduction

  • Automated creativity evaluation has focused mainly on simpler tasks such as AUT and TTCT, while task-specific models require costly training resources and annotated datasets.Complex multi-step creativity evaluation remains underexplored.Source: §1 · p. 1
  • CGPST (Wang et al., 2026b) requires scenario-grounded, interdependent steps and combines subjective outputs with wide scoring ranges, while direct LLM judging suffers low agreement and verbosity and leniency bias.Some dimensions use 10-level scoring, further increasing scoring instability.Source: §1 · p. 1
  • CreaEval separates analysis, where a SoT-LLM structures responses with cross-step memory, from rubric-based judging using extracted evidence rather than raw responses.The two phases are Memory-augmented Analysis and Evidence-based Judging.Source: §1 · p. 2
  • The decoupled design constrains judgments with structured evidence, improving scoring accuracy and stability while mitigating verbosity and leniency biases.Further analysis reports enhanced scoring stability across dimensions and reduced verbosity and leniency bias.Source: §1 · p. 2
  • 0.64 QWK human–LLM agreement on CGPST outperformed the second-best supervised-training baseline by 0.17.The reported metric is quadratic weighted kappa (QWK).Source: §1 · p. 2

2 Related Work

Traditional creativity benchmarks are usually single-step, while CGPST evaluates creative problem-solving through six interdependent steps. Prior LLM-based evaluation results on simpler tasks suggest promise, but direct judging remains challenging for CGPST.

2.1 Creativity Tasks

  • CGPST extends single-step AUT and TTCT tasks with scenario-grounded creative problem-solving across six interdependent steps, making it significantly more complex.AUT asks for novel uses of a common object, while TTCT uses open-ended scenarios.Source: §2.1 · p. 2
  • CreaEval separates evaluation into memory-augmented analysis that extracts stepwise evidence and judging that scores this evidence without raw responses.Judge-LLMs use the extracted evidence and predefined rubrics.Source: Figure 1 · p. 3

2.2 LLM-as-a-Judge for Creativity Evaluation

  • Lu et al. (2024) report 0.49 Kendall’s τ human–LLM agreement on AUT, while direct few-shot judging of CGPST reaches only 0.31 PCC.The AUT result across four dimensions exceeds the reported 0.39 inter-human agreement; these studies motivate more reliable evaluation for complex multi-step tasks.Source: §2.2 · p. 3

3 Methodology

CreaEval separates analysis from judging to address the loss of intermediate evidence in direct raw-response-to-score evaluation. Its two phases extract structured evidence with cross-step memory, then score using that evidence rather than raw responses.

3 Methodology

  • CreaEval replaces direct raw-response-to-score judging with two sequential phases: Memory-augmented Analysis and Evidence-based Judging.The framework is designed to capture intermediate evidence that direct judging may miss.Source: §3 · p. 3

3.1 Phase 1: Memory-augmented Analysis

  • SoT-LLM progressively extracts dimension-specific evidence from each step’s scenario, response, and evaluation dimensions, while passing a summary memory forward to preserve cross-step dependencies.The progressive process is intended to improve the reliability and consistency of extracted evidence.Source: §3.1 · pp. 3–4

3.2 Phase 2: Evidence-based Judging

  • Judge-LLM scores all steps and dimensions in one pass from aggregated evidence, rubrics, and scenario—not raw responses—assigning 6/10 for weak Step-2 Adequacy evidence.The example illustrates evidence-based scoring: “low importance” and “a minor issue” support the relatively low score.Source: §3.2 · p. 4

4 Experiment

The experiments center on CGPST, a complex six-step creativity benchmark, and use AUT and TTCW to assess generalizability. They compare CreaEval with multiple evaluation approaches and report QWK consistency results.

4.1 Challenging Dataset

  • CGPST contains 200 samples across 10 scenarios, with six interdependent steps, multiple scoring dimensions, and calibrated scores from two human evaluators whose inter-rater reliability is 0.84.Its subjectivity, cross-step dependencies, fine-grained scoring, and wide score ranges make it challenging for automated evaluation.Source: §4.1 · p. 4
  • AUT evaluates originality in single-turn alternative-use generation, while TTCW scores creative stories on fluency, flexibility, originality, and elaboration; CGPST remains the primary benchmark.AUT and TTCW extend evaluation beyond CGPST’s procedural, multi-step structure.Source: §4.1 · p. 4

4.2 Baselines

  • Table 2 presents CGPST QWK consistency results across methods and dimensions, with an AVG column and significance testing against the second-best method.For AUT and TTCW, structure-based baselines are omitted; TTCW also includes a train-free reference-based approach using human-written stories, and Table 3 reports QWK results.Source: Table 2 · p. 5, §4.2 · p. 5, Table 3 · p. 6
  • Baselines include direct scoring, CoT, ToT, GoT, TaT, SFT, and SaMer; TaT is an undecoupled structured-representation variant of CreaEval.SFT uses a 7:3 train-test split; SaMer adaptively identifies and weights dimensions by scenario.Source: §4.2 · p. 5

4.3 Experimental Setup

  • The setup uses four Judge-LLMs across methods, qwen3.6-plus as the SoT-LLM, and temperature 0.2; evaluation reports PCC, QWK, and ICC.The Judge-LLMs are qwen3.6-plus, gpt-5.4, deepseek-v4-pro, and gemini-3.1-pro.Source: §4.3 · p. 5

5 Results

CreaEval outperforms the baselines on CGPST and simple creativity tasks. Its results also show stronger performance across challenging dimensions, consistency across judges, and the importance of its key components.

5.1 Main Results

  • CreaEval reaches QWK 0.64 on CGPST, ahead of SFT (0.47) and training-free methods (0.2–0.25), and outperforms baselines on simple creativity tasks.The paper reports that QWK, PCC, and ICC show similar trends.Source: §5.1 · p. 6
  • On QWK, most training-free methods score below 0.2 on wide-range Steps 2/4; Step 5 is 0.08–0.46 versus CreaEval’s 0.94, while Step 6 is below 0.2 versus CreaEval’s ~0.5.The authors attribute weak training-free performance to retaining redundant information from earlier steps without effective evidence extraction, which disrupts later evaluations.Source: §5.1 · p. 6
  • CreaEval performs nearly identically across four Judge-LLMs, whereas training-free methods fluctuate across judges.The paper attributes this stability to using structured evidence instead of raw responses, reducing the influence of subjective variation and supporting model-agnostic evaluation.Source: §5.1 · p. 6, Figure 2 · p. 6

5.2 Ablation Study

  • Ablations lowered QWK to 0.49 (23%↓) without memory, 0.24 (63%↓) without evidence, 0.21 (67%↓) without decoupling, and 0.01 (98%↓) without step-wise extraction.The variants remove cross-step memory, structured evidence, separation of analysis and judging, or iterative step-by-step extraction, respectively; the authors link the all-at-once collapse to excessive input harming evidence extraction.Source: Table 4 · p. 7, §5.2 · p. 7

6 Discussion

The discussion evaluates CreaEval’s robustness, bias mitigation, scoring stability, inference cost, and case-study behavior beyond scoring accuracy.

6.2 Bias Mitigation in CreaEval

  • CreaEval aligns leniency-bias scores more closely with human annotations and alone reduces length-score correlation (-5.81%); CoT, GoT, and ToT worsen verbosity bias, while TaT rises (+0.82%).The leniency analysis examines Originality of Step-3; the verbosity analysis compares Step-6 response length with Development scores.Source: §6.2 · p. 7, Figure 3 · p. 8, Figure 5 · p. 8

6.1 Impact of SoT-LLM in CreaEval

  • QWK agreement stays above 0.6 across SoT-LLM and Judge-LLM combinations, suggesting CreaEval does not depend on a specific model pairing.The authors characterize it as a general and robust evaluation framework.Source: §6.1 · p. 7, Figure 4 · p. 8

6.3 Scoring Stability Analysis

  • CreaEval has lower inter-Judge variance on Correctly Used in Step-5, with shared extracted evidence helping four Judge-LLMs produce more consistent scores.The analysis computes score variance across four Judge-LLMs for each sample and dimension on CGPST.Source: §6.3 · pp. 7–8, Figure 6 · p. 8

6.5 Case Study

  • In the Step-2 case study, TaT produces overly positive evaluations, whereas CreaEval’s evidence-grounded judging mitigates over-optimistic scoring.CreaEval separates evidence extraction from judging, unlike the coupled TaT approach.Source: §6.5 · p. 8, Figure 7 · p. 9

6.4 Inference Cost Analysis

  • CreaEval costs substantially less than GoT and ToT while maintaining strong performance; GoT and ToT incur substantial overhead from expanded reasoning paths.The comparison covers inference time and token consumption, reported at the 50th and 95th percentiles.Source: §6.4 · p. 8, Table 5 · p. 9

7 Conclusions

CreaEval separates creativity evaluation into memory-augmented analysis and evidence-based judging, and the paper reports strong human-judgment alignment, greater scoring stability, and reduced verbosity and leniency biases.

7 Conclusions

  • Across four LLMs, CreaEval strongly aligns with human judgments and consistently outperforms all baselines.The conclusion reports these results across the evaluated models.Source: §7 · p. 9
  • CreaEval decouples CGPST evaluation into SoT-LLM analysis that structures multi-step responses using cross-step memory and Judge-LLM scoring based only on that evidence.The Judge-LLM does not access the raw responses.Source: §7 · p. 9
  • Further analysis finds that this decoupled design improves scoring stability and reduces verbosity and leniency biases in LLM-as-a-Judge.The conclusion frames these findings as insights for evaluating subjective creativity tasks.Source: §7 · p. 9

Limitations · A CGPST Benchmark Details · B Baseline Implementation Details

CreaEval’s limitations are its simple cross-step memory and reliance on the only publicly available multi-step creativity benchmark. The accompanying sections describe CGPST and specify implementations for prompting and trained baselines.

Limitations

  • CreaEval uses rule-based cross-step memory and is evaluated only on CGPST, the only publicly available multi-step creativity benchmark identified by the authors.The authors propose exploring hierarchical memory and evaluating additional datasets when they become available.Source: Limitations · p. 9

A CGPST Benchmark Details

  • CGPST evaluates contextualized, six-step problem-solving through process-oriented, multidimensional assessment, making it more complex than AUT and TTCT.Its steps are interdependent and progress toward resolving a real-world problem; each step is evaluated across multiple dimensions.Source: A CGPST Benchmark Details · p. 12
  • The 200 CGPST samples comprise 10 scenarios with 20 six-step responses each, annotated with validated scores from two human experts.Table 6 overviews the 10 future scenarios.Source: A CGPST Benchmark Details · p. 12, Table 6 · p. 12
  • Across six dependent steps, CGPST moves from identifying challenges to selecting a problem, generating solutions, ranking them, and developing an action plan.The task also asks for five evaluation criteria before solutions are ranked and the best is selected.Source: A CGPST Benchmark Details · p. 12

B Baseline Implementation Details

C Agreement Metric Details · D Complete Prompts · E LLM Details

These appendices document how human–LLM agreement is computed and how CGPST responses are converted into structured evidence and scores. They also specify evidence extraction and scoring structures, constrain Judge-LLM output, and note the use of four LLMs through official APIs.

C Agreement Metric Details

  • Agreement is averaged across separate comparisons with Human A and Human B; PCC measures trend correlation, QWK ordinal agreement, and ICC inter-rater consistency.The method avoids comparing LLM scores with the averaged human score, which could conceal disagreement with both annotators.Source: C Agreement Metric Details · p. 13

SoT-LLM System Prompts

  • The SoT-LLM is prompted to extract responses into a predefined schema as an objective structured-information model for CGPST.The task proceeds through six sequential steps, from identifying challenges to developing an action plan.Source: SoT-LLM System Prompts · p. 18

D.2 SoT-LLM User Prompts

  • SoT-LLM user prompts supply the current step, scenario, schema, dimension descriptions, and original response text as inputs.The prompt variables are {step_num}, {future_scenario}, {step_schema}, {score_dimensions_wo_rubrics}, and {raw_text}.Source: §D.2 · p. 18

SoT-LLM User Prompts

  • The SoT-LLM must extract verifiable evidence strictly from the response, use None when dimension evidence is absent, and summarize responses for later-step context.It must follow the supplied schema and avoid external information, unsupported inferences, or fabricated evidence.Source: SoT-LLM User Prompts · pp. 18–19

Judge-LLM System Prompts

  • The Judge-LLM is framed as an objective evaluator of CGPST responses using structured SoT output across the task’s six sequential steps.Those steps range from challenge identification through action-plan development.Source: Judge-LLM System Prompts · p. 19

D.4 Judge-LLM User Prompts

  • Judge-LLM user prompts provide the future scenario, concatenated SoT results from all six steps, scoring dimensions with rubrics, and a JSON output template.The scoring dimensions and rubrics span all steps.Source: §D.4 · p. 19

Judge-LLM User Prompts

  • The Judge-LLM is instructed to score the complete six-step response using the provided SoT output, dimensions, score ranges, and required format.The user prompt includes the scenario, structured result, and scoring-dimension descriptions.Source: Judge-LLM User Prompts · p. 20
  • Judge-LLM instructions require independent dimension scoring from the SoT result, exact scoring ranges, strict JSON, and numeric scores only.The output must retain the specified field names and include no extra fields or explanations.Source: Judge-LLM User Prompts · p. 20

D.5 SoT Schema for SoT-LLM

  • SoT extracts evidence item by item for Steps 1, 3, and 4, by dimension for other steps, then concatenates evidence from all steps for the Judge-LLM.The concatenated evidence is passed as the {sot_output} field.Source: §D.5 · p. 20

D.6 Score Template for Judge-LLM

  • The Judge-LLM score template is shared across baselines, with step-specific subtemplates for step-wise methods and summary-level aggregation for item-scored steps.Flexibility is calculated across distinct categories for Steps 1 and 3, so it appears only in the summary field rather than as an item-level score.Source: §D.6 · p. 21

E LLM Details

  • Table 7 summarizes the four LLMs used in the experiments, which were conducted via official APIs.Source: E LLM Details · p. 22

F Temperature Settings · G Exploration on Evidence-Based Supervised Fine-Tuning

CreaEval uses temperature 0.2 for both evidence extraction and judging, which the experiments associate with the most stable outputs. Evidence-score fine-tuning improves over Vanilla SFT, while CreaEval remains slightly more accurate without extra training.

F Temperature Settings

  • The temperature pilot samples 10 CGPST responses, selects Steps 1 and 2, and compares settings 0.2, 0.5, and 0.8.Lower temperatures are expected to yield more consistent outputs, while higher temperatures promote diversity.Source: F Temperature Settings · p. 22
  • Evidence extraction is repeated five times per response and dimension; Self-BLEU measures lexical overlap and BERTScore measures semantic similarity, with higher values indicating greater stability.The repeated evidence outputs for each dimension are compared across runs.Source: F Temperature Settings · p. 22
  • At temperature 0.2, all four SoT-LLMs achieve their highest semantic and lexical similarity, so CreaEval sets evidence extraction temperature to 0.2.Similarity across repeated extraction runs is evaluated using Self-BLEU and BERTScore.Source: F Temperature Settings · p. 22
  • Judge-LLM score variance stays below 0.2 at every tested temperature and is lowest for all four models at 0.2, which is selected for judging.Each response is scored five times using evidence generated at temperature 0.2.Source: F Temperature Settings · p. 23

G Exploration on Evidence-Based Supervised Fine-Tuning

  • SFT_Evidence fine-tunes on extracted evidence-score pairs from CreaEval’s judging phase using the same 7:3 train-test split as Vanilla SFT.Vanilla SFT instead maps raw multi-step responses directly to rubric scores.Source: G Exploration on Evidence-Based… · p. 23
  • Average QWK is 0.632 for SFT_Evidence versus 0.4665 for Vanilla SFT, while train-free CreaEval reaches 0.6388.CreaEval achieves superior accuracy without extra supervised fine-tuning or training overhead.Source: G Exploration on Evidence-Based… · p. 23

H Complete Results

The section reports QWK results across methods and dimensions in Table 2, with corresponding PCC and ICC results in Tables 12 and 13.

H Complete Results

I PCC and ICC Results for CreaEval Robustness Analysis · J Heatmap Distributions across Dimensions · K Violin Plots of Inter-Judge Variance across Dimensions

Across robustness and bias analyses, CreaEval performs consistently across SoT-LLMs, mitigates leniency bias, and reduces inter-judge scoring variance. Its evidence-grounded judging also yields identical scores when the evidence determines the Step-1 category count.

I PCC and ICC Results for CreaEval Robustness Analysis

  • PCC values exceeded 0.65 and ICC values exceeded 0.6 across SoT-LLM settings, indicating CreaEval does not depend on a specific LLM.The results support its use as a general, robust evaluation framework.Source: I PCC and ICC Results for CreaEval… · p. 23

J Heatmap Distributions across Dimensions

  • Across representative dimensions, CreaEval consistently alleviated leniency bias compared with other training-free methods.The analysis compares human scores against methods on Originality of Step-3.Source: J Heatmap Distributions across… · p. 24
  • In every Step-6 dimension, CreaEval produced a more balanced score distribution, demonstrating stronger alignment with human annotations.Step-6 requires holistic integration of prior steps to generate a final action plan.Source: J Heatmap Distributions across… · p. 24

K Violin Plots of Inter-Judge Variance across Dimensions

  • Across representative dimensions, CreaEval consistently showed lower inter-judge variance, with its four Judge-LLMs producing more similar scores.The analysis measures inter-judge variance on Correctly Used of Step-5.Source: K Violin Plots of Inter-Judge Variance… · p. 24
  • Decoupling analysis from judging enables evidence-grounded scoring that constrains plausible scores and enhances scoring stability.This two-phase design is given as the explanation for CreaEval’s lower inter-judge variance.Source: K Violin Plots of Inter-Judge Variance… · p. 24
  • On Flexibility of Step-1, CreaEval had no variance because evidence specified each challenge’s category and all four Judge-LLMs counted categories identically.The corresponding violin plot is empty.Source: K Violin Plots of Inter-Judge Variance… · p. 24

L Case Studies for Remaining Steps

Case studies for Steps 1 and 3–6 show CreaEval distinguishing rubric-relevant evidence from unsupported interpretation and producing judgments that better align with human judges. The Step-5 example highlights that separating analysis from scoring can improve the final score even when both methods identify the same errors.

Step-5

  • In Step-5, CreaEval scored the response correctly at 1 while TaT scored it incorrectly at 5, despite both detecting four scoring-matrix errors.The case illustrates that correct analysis alone did not prevent TaT from making an incorrect final judgment; CreaEval mapped the detected violations to the rubric requirement.Source: L Case Studies for Remaining Steps · p. 24, Figure 32 · p. 32

Step-1

  • In Step-1, CreaEval distinguished contextual relevance from unsupported overinterpretation, aligning more closely with human judges on limited elaboration and modest originality than TaT.TaT gave high scores for the richness and novelty of abstract concepts, while CreaEval grounded its analysis in the response’s contextual support.Source: Figure 29 · p. 30

Step-3

  • In Step-3, CreaEval identified missing concrete mechanisms for social integration and gave more conservative elaboration and originality scores aligned with human judgments.TaT emphasized coherence and technical framing, while CreaEval examined how the proposal would prevent filter bubbles and move online interaction toward meaningful offline integration.Source: Figure 30 · p. 31

Step-4

  • In Step-4, CreaEval recognized that the criterion’s objectives were overly narrow relative to the task’s community-wide fairness goal, aligning more closely with human judgments than TaT.TaT focused on youth participation as a relevant target, while CreaEval noted the criterion’s focus on both the “shortest time” and “highest participation rate.”Source: Figure 31 · p. 31

Step-6

  • In Step-6, CreaEval gave more conservative evaluations aligned with human judgments by noting limited global scalability, insufficient human-centered details, and missing implementation specifics.The expected impacts included a 40% reduction in concentration, a 50% reduction in bycatch rate, and USD 200 million in output value, indicating a significant positive contribution to the future scenario.Source: Figure 33 · p. 32

Related papers

Loading 2609.03432v1…