Source-linked AI summary
How do Humans and Language Models Reason About Creativity? A Comparative Analysis
Antonio Laverghetta Jr., Tuhin Chakrabarty, Tom Hope, Jimmy Pronchick, Krupa Bhawsar, Roger E. Beaty
TL;DR
Human and AI creativity judgments are increasingly used in science and engineering, but how evaluators reach those judgments remains poorly understood. The paper analyzes expert and LLM ratings and explanations, finding that humans and models associate originality with different facets, even as LLM predictions better match ground truth.
Problem
Although LLMs can predict human creativity assessments, little is known about the features they prioritize or whether their evaluation strategies align with human experts.
Source: Introduction · pp. 1–2Method
The studies analyze human and LLM originality evaluations of STEM design solutions, including written explanations and ratings of uncommonness, remoteness, and cleverness, with and without examples.
Source: Introduction · p. 2, Methods · p. 3Results
Human judges associated originality more strongly with cleverness, whereas LLMs associated it more strongly with remoteness and uncommonness despite more accurate predictions of ground-truth originality.
Source: Discussion · pp. 4–5, Conclusion · p. 6Takeaways & Limitations
Understanding these discrepancies remains important for automated creativity scoring, given current LLMs’ strong coarse-grained scoring and different facet correlations.
Source: Conclusion · p. 6The study varied neither prompt instructions nor example sets and presentation order, so some findings may reflect prompt structure or example selection and order.
Source: Conclusion · p. 6
Abstract
from arXiv · showhide
Creativity assessment in science and engineering is increasingly based on both human and AI judgment, but the cognitive processes and biases behind these evaluations remain poorly understood. We conducted two experiments examining how including example solutions with ratings impact creativity evaluation, using a finegrained annotation protocol where raters were tasked with explaining their originality scores and rating for the facets of remoteness (whether the response is "far" from everyday ideas), uncommonness (whether the response is rare), and cleverness. In Study 1, we analyzed creativity ratings from 72 experts with formal science or engineering training, comparing those who received example solutions with ratings (example) to those who did not (no example). Computational text analysis revealed that, compared to experts with examples, no-example experts used more comparative language (e.g., "better/worse") and emphasized solution uncommonness, suggesting they may have relied more on memory retrieval for comparisons. In Study 2, parallel analyses with state-of-the-art LLMs revealed that models prioritized uncommonness and remoteness of ideas when rating originality, suggesting an evaluative process rooted around the semantic similarity of ideas. In the example condition, while LLM accuracy in predicting the true originality scores improved, the correlations of remoteness, uncommonness, and cleverness with originality also increased substantially -- to upwards of $0.99$ -- suggesting a homogenization in the LLMs evaluation of the individual facets. These findings highlight important implications for how humans and AI reason about creativity and suggest diverging preferences for what different populations prioritize when rating.
Introduction
Creativity judgments are widely used in science and engineering, but the processes and biases behind human and AI evaluations remain poorly understood. This research compares how experts and LLMs evaluate STEM creativity and how examples affect their judgments.
Introduction
- Although LLMs can predict human creativity assessments, little is known about the features they prioritize or whether their strategies align with expert judgments.Understanding these processes matters as LLMs take on roles in scientific research and innovation.Source: Introduction · pp. 1–2
- Global originality scores can mask underlying evaluation processes, while originality is understood as combining uncommonness, remoteness, and cleverness.The relative contributions of these facets to expert judgments remain unclear.Source: Introduction · p. 1
- Earlier AUT studies found that judges can agree on creativity scores while differing in memory-related, temporal, and analytical language.Prior work examined idea-generation processes and used LIWC to analyze explanations of originality ratings.Source: Introduction · pp. 1–2
- The Design Problems Task elicits solutions to real-world STEM challenges that are rated for both originality and effectiveness.Its domains include ability differences and limitations, transportation and mobility, and social environments and systems.Source: Introduction · p. 2
- The study compares human experts’ and LLMs’ evaluation strategies for STEM creativity and tests whether example solutions and ratings change what they prioritize.It analyzes numerical originality ratings and written explanations alongside ratings of creativity facets.Source: Introduction · p. 2
Study 1: Human Creativity Evaluations
In Study 1, experts rated STEM design solutions for originality and three facets, with or without example ratings. Examples did not significantly improve originality accuracy, but facet relationships and explanation language differed by condition.
Study 1: Human Creativity Evaluations
- Experts rated DPT solutions for originality, uncommonness, remoteness, and cleverness, with originality explanations and facet ratings collected in both example conditions.The example condition used solutions scored 1, 3, and 5; the final sample included 37 participants in that condition and 35 in the no-example condition.Source: Study 1: Human Creativity Evaluations · p. 2, Methods · p. 3
Methods
- The analysis used linguistic markers to distinguish structured evaluation from intuitive justifications and to relate explanation language to the rated creativity facets.Markers included comparative and cleverness language; comparative language was linked to remoteness and uncommonness because those judgments often invoke prior solutions.Source: Methods · p. 3
Results
- Expert originality evaluations correlated moderately with originality in both conditions (no example r = 0.44; example r = 0.47), while examples increased originality–cleverness correlation and decreased originality–remoteness and originality–uncommonness correlations.Source: Results · pp. 3–4
- Experts without examples used more comparative and causal/analytical language than those with examples; differences were significant for both markers in GPT-4O ratings.GPT-4O found no significant condition differences for perceptual details, past/future language, or cleverness.Source: Results · p. 4
Discussion
- Without examples, experts may retrieve prior solutions from memory, placing more emphasis on remoteness and uncommonness; examples may leave resources available for other evaluation aspects.The authors note that remoteness and uncommonness judgments often involve comparisons, unlike assessing cleverness in isolation.Source: Discussion · p. 4
Study 2: LLM Creativity Evaluations
The study compares human and LLM creativity evaluations with and without examples, finding that examples improve LLM prediction of originality while strengthening associations among originality facets. LLM explanations also show less linguistic variety than human explanations.
Study 2: LLM Creativity Evaluations
- The LLM experiment was designed to compare model and human evaluation strategies using parallel analyses of creativity ratings and written explanations.The authors motivate this comparison by noting that examples may improve model accuracy while also biasing evaluation.Source: Study 2: LLM Creativity Evaluations · p. 4
Methods
- The study used Claude 3.5 Haiku and GPT-4O-MINI to rate originality, cleverness, remoteness, and uncommonness, with and without the examples shown to human raters.The models were selected as cost-effective smaller variants; both conditions used the same datasets as Study 1.Source: Methods · p. 4
Results
- LLM originality-ground-truth correlations rose from r = 0.6, 0.67 without examples to r = 0.74, 0.76 with examples; models rarely used 5 and mostly rated responses 2.Figure 2 summarizes GPT-4O-MINI facet correlations; Figure 3 compares human and model originality scores.Source: Results · p. 5, Figure 2 · p. 5, Figure 3 · p. 5
- LLMs prioritized remoteness and uncommonness over cleverness when predicting originality without examples; with examples, cleverness became a stronger predictor and all facet correlations increased.The weaker cleverness association in the no-example condition dissipated in the example condition.Source: Results · p. 5
Discussion
- LLM explanations had more skewed linguistic-marker distributions and less variability than human explanations, with some markers absent across all model explanations.The authors note that low temperature may partly contribute to this pattern, but exact duplicate explanations were uncommon.Source: Results · p. 5, Discussion · p. 5
- With examples, correlations between originality and each LLM-rated facet increased significantly, while the facets became more strongly intercorrelated than in human ratings.The authors interpret this homogenization as weaker construct validity despite stronger predictive validity.Source: Discussion · p. 5, Results · p. 5
Conclusion
The paper compares fine-grained creativity evaluations by human experts and LLMs, identifying differences in facet ratings and explanations. It concludes that greater LLM originality accuracy can coexist with homogenized facet judgments, while noting constraints on prompts and examples.
Conclusion
- Human and LLM evaluators differed in how originality related to creativity facets and in how they structured explanations, with potential validity implications for LLM scores.The authors report that LLMs more accurately predicted ground-truth originality while showing substantially different facet correlations.Source: Conclusion · p. 6
- Prompt variations and multiple example sets or presentation orders were not tested, leaving open whether prompt structure or example selection influenced some findings.The authors say these constraints may have biased facet judgments, though they do not believe they fully explain homogenization.Source: Conclusion · p. 6
- The authors frame fine-grained assessment of human and AI creativity judgments as important for developing AI that evaluates creativity accurately and fairly.They call for similar fine-grained assessments in other domains.Source: Conclusion · p. 6
LLM Prompts
The prompts for linguistic-marker analysis followed a minimal-context approach, and the paper presents separate prompts for five markers and for Study 2.
LLM Prompts
- The linguistic-marker prompts cover past/future language, perceptual details, causal/analytical language, comparative language, and cleverness.The prompts are shown in Figures 5–9, respectively.Source: LLM Prompts · p. 9
- The authors structured prompts for LLM analysis of explanations similarly to Rathje et al. (2024), who gave models minimal context when rating LIWC categories.Source: LLM Prompts · p. 9
- The Study 2 prompt is shown separately in Figure 10.Source: LLM Prompts · p. 9
Results
The LLMs’ facet ratings became more aligned in the example condition, while cleverness’s relationship to originality varied by model. Claude-3.5-Haiku showed similar but often smaller correlation shifts than GPT-4O-MINI.
Results
- Agreement among all facet pairs rose sharply in the example condition, indicating homogenization.The trend appeared in both Pearson correlations and kappa agreement statistics.Source: Results · p. 9
- Cleverness was GPT-4O-MINI’s weakest predictor of originality, but the opposite was true for Claude-3.5-Haiku.The passage reports this model difference for kappa agreement trends.Source: Results · p. 9
- Claude-3.5-Haiku’s facet-correlation trends resembled GPT-4O-MINI’s, though correlations were often less strong.Figure 11 summarizes Haiku’s pairwise Likert-rating correlations; o = originality, c = cleverness, u = uncommonness, and r = remoteness.Source: Results · p. 9, Figure 11 · p. 9
- Figures 12 and 13 report kappa agreement statistics for GPT-4O-MINI and Claude-3.5-Haiku, respectively, in both conditions.The figures label pairwise ratings as originality, cleverness, uncommonness, and remoteness.Source: Results · p. 9, Figure 12 · p. 9, Figure 13 · p. 9
Additional Explanation Results
Analyses of explanation language found condition differences in several markers but not perceptual details. The supplementary figures show the prompts and model analyses used for this work.
Additional Explanation Results
- Figure 15 presents Claude-3.5-Haiku results and Claude-3.5-Sonnet ratings of human and GPT-4O-MINI explanations.The supplied results report significant condition differences for past/future, comparative, and cleverness language, but no perceptual-detail difference.Source: Additional Explanation Results · p. 9
- Both evaluators found condition differences in past/future, comparative, and cleverness language, but not in perceptual details.The comparisons were for Claude-3.5-Haiku explanations, rated by Claude-3.5-Sonnet and GPT-4O.Source: Additional Explanation Results · p. 9
- The prompts cover past/future language, perceptual details, causal/analytical language, comparative language, and cleverness.The prompt figures fill [RESPONSE] with participant explanations.Source: Figure 5 · p. 10, Figure 6 · p. 10, Figure 7 · p. 10
Explanation Qualitative Analysis
Human explanations tended to be shorter and more varied, while LLM explanations were more formulaic and referred more directly to prior ideas. LLM explanations sometimes discussed individual creativity facets despite being prompted to focus on originality.
Explanation Qualitative Analysis
- Human explanations tended to be shorter, more egocentric, and more variable in structure and focus than LLM explanations.Table 1 provides sample explanations from humans and GPT-4O-MINI for the same prompt.Source: Explanation Qualitative Analysis · p. 12, Table 1 · p. 13
- LLM explanations were more formulaic and made more direct references to prior ideas.The examples compare human and GPT-4O-MINI explanations for the same design problem.Source: Explanation Qualitative Analysis · p. 12, Table 1 · p. 13
- LLMs sometimes addressed cleverness, remoteness, or uncommonness even though the prompt asked them to focus on originality.The authors say this supports the idea that LLMs homogenize these facets with originality.Source: Explanation Qualitative Analysis · p. 12
- Figure 14 reports human and GPT-4O-MINI rating distributions for cleverness, remoteness, and uncommonness.Figure 15 includes additional linguistic-marker comparisons across human and LLM explanations and raters.Source: Figure 14 · p. 14, Figure 15 · p. 15