PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

Xiangyu Wang, Jin Wu, Xiaoyu Li, Chanjin Zheng, Yifeng Zhou

arXiv:2609.03432v1cs.CL

TL;DR

Automated evaluation is unreliable for complex creativity tasks because CGPST combines interdependent steps, subjectivity, and broad scoring ranges. CreaEval separates memory-augmented evidence analysis from evidence-based judging, and reports stronger human-LLM alignment with reduced scoring biases. Its evaluation is limited by reliance on a simple rule-based memory mechanism and experiments conducted only on CGPST.

  • Problem

    CGPST challenges automated evaluation through interdependent steps, subjective outputs, and broad scoring ranges that increase judgment instability.

  • Method

    CreaEval uses a SoT-LLM with cross-step memory to extract structured evidence, then Judge-LLMs score that evidence without accessing raw responses.

  • Results

    CreaEval achieves 0.64 QWK human-LLM agreement on CGPST, outperforming the second-best baseline by 0.17.

  • Takeaways & Limitations

    The decoupled design improves scoring stability and mitigates verbosity and leniency biases in LLM-as-a-Judge.

  • Takeaways & Limitations

    Experiments use a simple rule-based memory mechanism and are conducted exclusively on the currently available CGPST multi-step dataset.

Abstract

from arXiv · show

Automated evaluation of creativity tasks remains challenging for LLM-as-a-Judge, as LLM is susceptible to biases such as verbosity bias and leniency bias. Such limitations are particularly evident in Contextually-Grounded and Procedurally-Structured Tasks (CGPST), a complex multi-step creativity task where inter-step dependencies, highly subjectivity, and wide scoring ranges lead to more unstable and biased judgments. Existing approaches either rely on task-specific training or directly apply LLM-as-a-Judge, both of which struggle to ensure reliable evaluation under such complexity. To bridge these gaps, we propose CreaEval, an automated creativity evaluator for CGPST that decouples typical LLM-as-a-Judge into analysis and judging. Correspondingly, CreaEval involves two critical phases: Memory-augmented Analysis, a SoT-LLM converts multi-step responses into structured evaluation evidence, incorporating cross-step memory; and Evidence-based Judging, a Judge-LLM uses the extracted evidence for judging without accessing raw responses. Comprehensive experiments show that CreaEval achieves an average performance improvement of 22.74% over the second-best baselines across CGPST and two classic simple creativity tasks, demonstrating its generalizability. The code is available at https://github.com/Jaong/CreaEval.

1 Introduction

Automated evaluation is especially difficult for complex, subjective, multi-step creativity tasks because interdependent steps and broad scoring ranges destabilize judgments. CreaEval addresses this by separating structured analysis from evidence-based judging and improves agreement while reducing identified LLM-as-a-Judge biases.

  • Existing approaches: Existing approaches use either costly task-specific training with annotated data or training-free LLM-as-a-Judge.The two approaches differ in training requirements but both are considered in the context of complex creativity evaluation.
  • Challenges: CGPST requires evaluating multiple interdependent steps across dimensions in subjective, open-ended outputs, with some dimensions using 10-level scoring.These properties increase the difficulty and instability of automated evaluation.
  • CreaEval: CreaEval decouples evaluation into Memory-augmented Analysis and Evidence-based Judging.A SoT-LLM structures response evidence with cross-step memory, while Judge-LLMs score using that evidence and predefined rubrics.
  • Results: 0.64 QWK human-LLM agreement on CGPST exceeds the second-best supervised-training baseline by 0.17.The reported result concerns average agreement on the CGPST benchmark.
  • Results: CreaEval’s decoupled design improves scoring stability across dimensions and mitigates verbosity and leniency biases.The paper presents these findings as further analysis of the proposed evaluation design.

2 Related Work

Traditional creativity tasks are typically single-step and structurally simple, whereas CGPST models more complex, scenario-grounded creative processes. CreaEval responds with a two-phase framework that extracts structured evidence before rubric-based judging.

  • Creativity tasks: AUT and TTCT are traditional creativity tasks that are generally single-step and structurally simple.Their structure limits how fully they capture complex creative processes.
  • CreaEval: CreaEval uses Memory-augmented Analysis to organize raw responses into structured intermediate evidence with cross-step coherence.The analysis phase uses an SoT-LLM and memory mechanism.
  • CreaEval: Judge-LLMs assign scores from extracted evidence and predefined rubrics without accessing raw responses.This is the second phase of the CreaEval framework.
  • Prior work: Prior creativity-evaluation research includes LLM scoring for TTCT-inspired datasets and LLM-as-a-Judge evaluation of AUT.The cited AUT work reported average human–LLM agreement of 0.49 Kendall’s τ across four dimensions.

3 Methodology

CreaEval separates creativity evaluation into structured evidence extraction and evidence-based scoring. Its memory mechanism preserves dependencies across CGPST steps, while judging uses aggregated evidence rather than raw responses.

  • Existing LLM-as-a-Judge methods map raw multi-step responses directly to scores, limiting their ability to capture intermediate evidence.
  • Memory-augmented Analysis: Memory-augmented Analysis incrementally processes each step using the scenario, response, dimensions, and prior memory.
  • Memory-augmented Analysis: SoT-LLM represents each step’s extracted evidence as dimension–evidence pairs, while progressive extraction avoids processing all subjective information simultaneously.
  • Evidence-based Judging: Evidence-based Judging supplies aggregated evidence, rubrics, and the scenario to Judge-LLM, which generates scores without accessing raw responses.
  • Evidence-based Judging: For Step-2 Adequacy, evidence such as “low importance” and “a minor issue” leads Judge-LLM to assign a score of 6 out of 10.

4 Experiment

The experiments evaluate CreaEval on complex six-step CGPST tasks and two simpler creativity benchmarks, comparing it with direct, reasoning-structured, supervised, and scenario-aware baselines.

  • Datasets: Experiments use CGPST, a future-scenario benchmark requiring six interdependent steps with multiple scoring dimensions.
  • Datasets: AUT evaluates novel object uses on Originality, while TTCW evaluates creative writing under a different task structure.
  • Evaluation: CGPST comparisons report consistency using QWK across methods and dimensions, with AVG summarizing overall performance and asterisks marking significant improvements over the second-best method.
  • Baselines: Baselines include Direct Score, CoT, ToT, GoT, TaT, SFT, and SaMer, covering direct scoring, structured reasoning, supervised training, and scenario-aware evaluation.
  • Evaluation: Four Judge-LLMs are used across methods, while qwen3.6-plus serves as the SoT-LLM for structured evidence extraction.

5 Results

CreaEval achieves the strongest reported consistency across CGPST and simpler creativity tasks, with improvements linked to structured evidence, memory, and decoupled judging. Ablations show that removing these components substantially degrades performance.

  • Main Results: 0.64 QWK is CreaEval’s CGPST performance, exceeding SFT at 0.47 and training-free methods at 0.2–0.25.
  • Main Results: CreaEval also achieves superior performance on AUT and TTCW, demonstrating generalizability beyond complex multi-step evaluations.
  • Dimension Analysis: Step-2 and Step-4 wide-range dimensions keep most training-free baselines below 0.2, whereas CreaEval reaches 0.94 on Step-5.
  • Dimension Analysis: In Step-6, training-free baselines fall below 0.2 while CreaEval maintains consistency around 0.5 on the holistic final action-plan task.
  • Robustness: CreaEval shows nearly identical performance across four Judge-LLMs, indicating stable evaluation behavior across judges.
  • Ablation Study: Removing memory yields QWK = 0.49, removing evidence yields QWK = 0.24, removing decoupling yields QWK = 0.21, and single-pass extraction yields QWK = 0.01.

6 Discussion

CreaEval is analyzed for robustness, bias mitigation, stability, efficiency, and qualitative behavior beyond its scoring accuracy. Across these analyses, structured evidence supports robust, stable, less biased judgments while retaining a better cost-performance balance than the most expensive reasoning methods.

  • Robustness: CreaEval maintains QWK above 0.6 across all tested SoT-LLM and Judge-LLM combinations.This indicates robustness without dependence on a specific LLM.
  • Bias Mitigation: CreaEval produces a more balanced score distribution aligned with human annotations, mitigating the high-score clustering shown by training-free methods.The comparison concerns Originality of Step-3 and addresses leniency bias.
  • Bias Mitigation: Only CreaEval reduces the response-length correlation with Development scores, by -5.81% relative to Direct Score.CoT, GoT, and ToT exacerbate verbosity bias, while TaT increases the correlation by +0.82%.
  • Scoring Stability: CreaEval yields lower inter-Judge variance because all Judge-LLMs score shared extracted evidence rather than raw responses.The variance analysis uses four Judge-LLMs across samples and dimensions.
  • Efficiency: CreaEval costs substantially less than ToT and GoT while maintaining strong performance.GoT and ToT incur substantial overhead from expanded reasoning paths, whereas CoT and TaT have relatively low consumption.
  • Case Study: A Step-2 case study shows TaT producing overly positive judgments, whereas CreaEval grounds scoring in structured evidence and reduces over-optimistic evaluations.The comparison illustrates the effect of coupling versus decoupling analysis and judging.

7 Conclusions

CreaEval is presented as a framework that decouples memory-augmented analysis from evidence-based judging for CGPST. The conclusion attributes improved scoring stability and reduced verbosity and leniency biases to this decoupled design.

  • Framework: CreaEval decouples CGPST evaluation into memory-augmented analysis and evidence-based judging.SoT-LLM converts multi-step responses into structured evidence, while Judge-LLM scores using that evidence.

Limitations

The paper’s scope is constrained by its memory mechanism and by the limited availability of public multi-step creativity benchmarks. The supplied benchmark and implementation details describe CGPST’s structure and the compared evaluation methods.

  • Limitations: CreaEval uses a simple rule-based memory mechanism tailored to CGPST’s fixed step dependencies.The authors identify hierarchical memory modules as a possible direction for more general settings.
  • Limitations: Experiments are conducted exclusively on CGPST because it is described as the only publicly available multi-step creativity benchmark dataset.The authors plan evaluation on additional datasets when they become available.
  • Benchmark Scope: CGPST contains 10 scenarios with 20 complete six-step response samples each, totaling 200 samples annotated by two human experts.The benchmark uses sequential, interdependent steps with process-oriented and multidimensional assessment.
  • Compared Methods: The evaluation compares direct scoring, sequential or tree-structured reasoning, graph refinement, tabular reasoning, supervised fine-tuning, and a scenario-aware scoring model.These methods differ in reasoning structure, training requirements, or scoring architecture.

C Agreement Metric Details

Agreement analysis compares model scores with separate human annotations using correlation, ordinal agreement, and inter-rater reliability measures. The section also supplies CGPST examples and extraction guidance requiring evidence to remain grounded in responses.

  • Agreement Computation: LLM agreement is computed separately against Human A and Human B, then averaged rather than compared with averaged human scores.This avoids a model score appearing accurate merely because it matches an average that neither rater assigned.
  • Agreement Metrics: PCC measures linear correlation between predicted scores and human annotations, capturing overall trend consistency.The formula uses predicted and human scores together with their respective means.
  • Agreement Metrics: QWK measures ordinal agreement while accounting for the degree of disagreement between raters.It is suited to discrete or ordinal scoring tasks.
  • Agreement Metrics: ICC measures the proportion of variance attributable to differences between subjects relative to total variance.It reflects consistency across quantitative raters.
  • Evidence Extraction: SoT extraction must produce fine-grained, verifiable evidence grounded in the response and use None when a dimension lacks relevant evidence.The schema is intended to support subsequent dimension-level scoring without fabricated information.

F Temperature Settings

The temperature study evaluates stability in evidence extraction and judging, finding that lower-temperature extraction is more consistent and motivating a setting of 0.2. Evidence-based scoring also substantially outperforms direct supervised scoring.

  • Temperature study: Three temperature settings—0.2, 0.5, and 0.8—were evaluated on repeated evidence extraction for randomly sampled CGPST responses.The pilot study analyzed Step-1 and Step-2 responses from 10 randomly sampled examples.
  • Extraction stability: At temperature 0.2, all four SoT-LLMs achieved their highest semantic and lexical similarity across repeated extractions.The study uses BERTScore for semantic similarity and Self-BLEU for lexical similarity.
  • Configuration: The SoT-LLM temperature was therefore set to 0.2 for subsequent evidence generation.
  • Evidence-based scoring: SFT_Evidence achieved 0.632 Average QWK versus 0.4665 for Vanilla SFT, approaching CreaEval at 0.6388.SFT_Evidence uses extracted evidence-score pairs, while CreaEval achieves its result without supervised fine-tuning.

H Complete Results

The complete-results section reports QWK across methods and dimensions, with PCC and ICC results provided as complementary consistency measures. Figures 8 and 9 summarize PCC and ICC results across different SoT-LLMs.

  • Overall results: QWK results are reported across all methods and dimensions in the main paper’s Table 2.
  • SoT-LLM comparison: Figures 8 and 9 report PCC and ICC results across different SoT-LLMs, respectively.
  • Consistency metrics: PCC results corresponding to the main QWK evaluation are presented in Table 12.
  • Consistency metrics: ICC results corresponding to the main QWK evaluation are presented in Table 13.

I PCC and ICC Results for CreaEval Robustness Analysis

CreaEval remains robust when different SoT-LLMs are substituted, with consistently strong PCC and ICC values. Additional analyses show reduced leniency bias, lower inter-Judge variance, and identical scores when evidence fixes the relevant category information.

  • Robustness: PCC values remain above 0.65 and ICC values exceed 0.6 across SoT-LLM settings.These results support CreaEval’s robustness without dependence on a specific SoT-LLM.
  • Leniency bias: Across representative dimensions, CreaEval consistently alleviates leniency bias relative to other training-free methods.The advantage is especially evident in Step-6, which integrates information from previous steps.
  • Scoring stability: CreaEval exhibits lower inter-Judge variance across representative dimensions than competing methods.The reported pattern indicates more similar scores among the four Judge-LLMs within CreaEval.
  • Scoring stability: On Step-1 Flexibility, all four Judge-LLMs produced identical scores because the extracted evidence supplied each challenge’s category.The corresponding variance is zero.

L Case Studies for Remaining Steps

Case studies across later steps show that CreaEval can map detected evidence to rubric requirements more conservatively and accurately than coupled analysis-and-judging methods. The examples cover subjective interpretation, optimization criteria, ranking-matrix errors, and impact or humaneness judgments.

  • Step-5: In Step-5, CreaEval maps four detected ranking-matrix column errors to the rubric and assigns score 1, whereas TaT assigns score 5.The matrix contains repeated and missing scores in four criterion columns.
  • Step-1: In Step-1, CreaEval distinguishes contextual relevance from unsupported overinterpretation, aligning more closely with human judgments than TaT.TaT emphasizes abstract concepts and assigns high scores despite limited elaboration and modest originality.
  • Step-3: In Step-3, CreaEval examines missing concrete mechanisms such as preventing filter bubbles and enabling offline social integration.Its more conservative scores align better with human judgments than TaT’s emphasis on coherence and technical framing.
  • Step-4: In Step-4, CreaEval identifies multiple optimization objectives and a criterion narrower than the task’s community-wide fairness goal.The example contrasts this analysis with TaT’s focus on youth participation.
  • Ranking outcome: The ranking example selects solution 3, which receives the highest total score of 34.The selected solution proposes edge-field planting training for farmers in Benin.
  • Impact: The impact example assigns score 5 to quantitatively specific reductions in concentration and bycatch alongside USD 200 million in output value.
  • Humaneness: The humaneness example assigns score 5 to biodiversity protection, local community participation, and environmental governance.
Loading 2609.03432v1…