Source-linked AI summary
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Harsh Kumar, Jonathan Vincentius, Ewan Jordan, Ashton Anderson
TL;DR
Whether LLM assistance changes unassisted human creativity remains unclear. Two preregistered experiments with 1,100 participants compared standard and coach-like LLM assistance with no assistance across divergent- and convergent-thinking tasks, including later unassisted rounds. Assistance improved performance during assisted tasks but did not improve subsequent unassisted performance.
Problem
Method
Two preregistered parallel experiments with 1,100 participants compared standard and coach-like LLM assistance with no assistance across divergent- and convergent-thinking tasks followed by unassisted tests.
Source: §1 · p. 2Results
LLM assistance improved performance during assisted tasks but did not improve subsequent unassisted performance; divergent-task exposure also sometimes reduced originality and idea diversity.
Source: §1 · p. 2Takeaways & Limitations
The findings suggest that the mode of LLM assistance matters and may shape long-term creative abilities.
Source: §6 · p. 14Coach-like guidance was more verbose and cognitively demanding than direct answers, potentially making exposure-condition comparisons unfair; future studies should control verbosity and standardize guidance.
Source: §5.3 · p. 14
Abstract
from arXiv · showhide
Large language models are transforming the creative process by offering unprecedented capabilities to algorithmically generate ideas. While these tools can enhance human creativity when people co-create with them, it's unclear how this will impact unassisted human creativity. We conducted two large pre-registered parallel experiments involving 1,100 participants attempting tasks targeting the two core components of creativity, divergent and convergent thinking. We compare the effects of two forms of large language model (LLM) assistance -- a standard LLM providing direct answers and a coach-like LLM offering guidance -- with a control group receiving no AI assistance, and focus particularly on how all groups perform in a final, unassisted stage. Our findings reveal that while LLM assistance can provide short-term boosts in creativity during assisted tasks, it may inadvertently hinder independent creative performance when users work without assistance, raising concerns about the long-term impact on human creativity and cognition.
1 Introduction
The introduction asks whether co-creating with LLMs affects people’s ability to create independently, and presents experiments finding that assistance may help during tasks but hinder later unassisted performance.
1 Introduction
- The introduction motivates the study by noting concerns that reliance on generative AI could impair independent creativity or homogenize thought, while AI could also stimulate creativity as a fresh partner.Because creativity is subjective and affected by prior knowledge, experience, and context, the effects of AI use in everyday settings are difficult to isolate.Source: §1 · p. 2
- The experiments examine divergent thinking—generating multiple unique ideas—and convergent thinking—refining ideas to select effective solutions—across two forms of LLM assistance and a no-assistance control.The standard LLM provided direct answers, while the coach-like LLM offered guidance and prompts intended to stimulate thinking.Source: §1 · p. 2
- Two preregistered parallel experiments with 1,100 participants compared standard LLM answers, coach-like guidance, and no assistance, testing performance during assisted rounds and after a delay in unassisted rounds.Participants were randomly assigned to one of the three conditions and completed the same type of creative task in exposure and test rounds.Source: §1 · p. 2
- LLM exposure did not improve unassisted divergent originality or fluency and sometimes reduced originality and idea diversity; convergent-task gains did not persist, and guided participants underperformed controls.The study also identifies persistent homogenization effects associated with LLM-generated strategies.Source: §1 · p. 2
- The paper reports skepticism toward LLM assistance in divergent tasks, benefits in convergent tasks, and design implications for tools that support creativity without undermining independent ability.The stated goal is to develop LLM-based creativity tools that enhance human creativity without weakening independent creative abilities.Source: §1 · pp. 2–3
2 Related Work
Related work frames creativity as both idea generation and refinement, while identifying an unresolved question: how repeated LLM assistance affects human divergent and convergent thinking, including unassisted performance.
2.1 Theories of Human Creativity
- Poincaré frames creativity as generation followed by refinement, while Boden distinguishes individual novelty (P-creativity) from historically novel ideas (H-creativity), operationalized here with AUT and RAT for divergent and convergent thinking.Divergent thinking is linked to idea generation; convergent thinking helps select and refine ideas toward solutions.Source: §2.1 · p. 3
2.2 LLMs as Tools for Creativity
- Prior work explores LLMs as task performers and creativity-support guides, but reports homogenized idea generation and convergence on limited ideas [2, 3, 58, 59].Some studies compare LLM and human performance or examine a single round of human idea generation with LLM support.Source: §2.2 · p. 3
2.3 Generative AI and Human Cognition
- Accounts describe Generative AI’s cognitive effects as polarized, from concerns about homogenization to potential creative growth, and distinguish short-term “steroids” from a capability-building “coach” [2].The “steroids, sneakers, and coach” metaphor describes different possible roles for AI in human cognition.Source: §2.3 · pp. 3–4
- The study addresses an empirical gap by testing both divergent and convergent human creativity across repeated LLM exposures and measuring residual effects in unassisted work, beyond LLM-output or single-exposure studies [40], [3].The authors contrast this design with prior work focused on LLM creative output or Human+AI creativity after a single exposure.Source: §2.2 · p. 3, §2.3 · p. 4
3 Experiment 1: Divergent Thinking
In Experiment 1, prior LLM assistance affected some divergent-thinking outcomes, but participants exposed to strategies had lower originality and more similar ideas during unassisted testing. Test-phase fluency did not differ significantly across conditions.
3.1 Experimental Design
- The 460 participants completed three two-minute AUT exposure rounds with no support, seven AI-generated ideas, or seven SCAMPER-based strategies, then completed an unsupported task for a new object.A one-minute Snake game separated exposure from the solo test; the objects were randomly selected from tire, pants, shoe, table, and bottle.Source: §3.1 · p. 4, §3.1.1 · pp. 4–6, §3.1.5 · p. 6
3.1.3 Dependent Variables.
- Outcomes measured idea originality, fluency, individual- and group-level diversity, and creative flexibility, with condition differences assessed using Kruskal-Wallis tests and corrected Dunn pairwise tests.Originality used a fine-tuned GPT-3 classifier; diversity and flexibility were calculated from SBERT idea embeddings.Source: §3.1.3 · p. 6, §3.1.4 · p. 6
3.2.1 Originality.
- H(2) = 9.14, p = 0.010: in the unassisted test, strategy-exposed participants scored lower in originality than controls (p = 0.009); other pairwise differences were nonsignificant.The List of Strategies group’s originality also declined from exposure to test, while the List of Ideas group showed no significant difference from either group.Source: Figure 4 · p. 7, §3.2.1 · p. 7
3.2.4 Individual- and Group-level Diversity of Idea Set.
- Strategy users produced more similar ideas within each round than controls during both exposure and unassisted testing, while the other pairwise comparisons were nonsignificant.In exposure, strategy users’ ideas were also more similar than those in the List of Ideas condition; the test-phase difference was significant only versus controls.Source: Figure 5 · p. 7, §3.2.4 · p. 8
- Simulated group-level idea diversity was similar across conditions during exposure, while strategy-exposed participants continued producing more similar ideas during unassisted testing.The authors describe this persistence of similarity as potential homogenization; they suggest participants may not have adopted the shared AI ideas in exposure.Source: Figure 5 · p. 7, §3.2.4 · p. 8
3.2.2 Fluency.
- H(2) = 1.14, p = 0.566: prior LLM exposure did not significantly affect the number of ideas participants generated in the unassisted test.During exposure, the List of Strategies group produced fewer ideas than controls (H(2) = 8.57, p = 0.014; pairwise p = 0.011).Source: §3.2.2 · p. 7
3.2.3 Creative Flexibility.
- Creative flexibility differed overall (H(2) = 6.31, p = 0.043), but no corrected pairwise comparison was significant; strategy users only tended to reuse ideas in the test (p = 0.052 vs. controls).Flexibility measured how dissimilar test-round ideas were from ideas generated during exposure.Source: §3.2.3 · p. 8
3.2.5 Subjective Measures and Perceptions.
- Across conditions, participants reported lower perceived creativity after the experiment; feelings toward AI were relatively stable, and the List of Ideas group found test ideas easier to generate.The List of Ideas group reported more than double the magnitude of change in feelings toward AI compared with the other groups.Source: §3.2.5 · pp. 8–9
4 Experiment 2: Convergent Thinking
Experiment 2 used the Remote Associates Test to examine whether prior direct-answer or guidance-style LLM assistance affected unassisted convergent thinking. Guidance participants performed worse than the no-LLM group in the test rounds, while participants in both AI conditions reported reduced creativity.
4.1 Experimental Design
- Participants solved RAT problems with assistance during three exposure rounds, then completed two unassisted test rounds; accuracy was the convergent-thinking measure.The Remote Associates Test asks participants to connect three presented words with a fourth within one minute.Source: §4.1 · p. 9
4.1.1 Conditions and treatment.
- Exposure conditions were No LLM, GPT-4o direct answers, or GPT-4o guidance that offered associated words without revealing the solution.After a one-minute Snake distractor, all participants attempted randomly selected RAT problems without AI.Source: §4.1.1 · p. 9
4.1.2 Stimulus.
- The study used 45 RAT questions across easy, medium, and hard levels, selecting easy and medium items for each round; a verbal-fluency task screened for English proficiency differences.Pilot testing identified the easy and medium items as appropriate for the crowdworker participant pool.Source: §4.1.2 · p. 10, §4.1.3 · p. 10
4.1.3 Verbal Fluency and Surveys.
- The study surveyed creativity, AI attitudes, perceived test difficulty, and exposure helpfulness, and its preregistered ANCOVA compared test accuracy across conditions while controlling for verbal-fluency word count.Tukey’s HSD was used for pairwise comparisons after a significant ANCOVA.Source: §4.1.3 · pp. 10–11, §4.1.4 · p. 11
4.2.1 Accuracy in the Test Rounds.
- During exposure rounds, LLM Guidance participants performed significantly worse than No LLM Assistance (mean difference −10.6; 95% CI: −18.6, −2.7; p = 0.005); condition had a significant effect, F(2, 1256) = 5.134, p = 0.006.LLM Answer did not differ significantly from No LLM Assistance (p = 0.682), while its comparison with LLM Guidance showed a nonsignificant trend (p = 0.064).Source: §4.2.1 · p. 11
4.1.4 Analysis.
- The experiment recruited 640 fluent-English participants in the US or UK through Prolific, with sample size set by power analysis for 80% power on the main preregistered test.Participants took about 10 minutes and were paid $1.30; 99% passed the attention check.Source: §4.1.5 · p. 11
4.2.2 Subjective Measures and Perceptions.
- Self-reported creativity fell after the experiment in every condition, with larger average declines in both LLM conditions and a guidance-condition decline more than twice that of No LLM Assistance.Attitudes toward everyday AI use stayed roughly constant, with a slight increase in the LLM Answer condition.Source: §4.2.2 · pp. 11–12, Figure 10 · p. 12
- Test rounds felt most difficult after LLM Guidance, while fewer participants found exposure helpful for test performance in the Answer and Guidance conditions than in No LLM Assistance.The helpfulness proportions were roughly 5% lower for LLM Answer and 7% lower for LLM Guidance; the question concerned whether exposure aided test performance, not whether AI itself was helpful.Source: Figure 10 · p. 12, §4.2.2 · p. 12
5 Discussion
LLM help can improve performance during exposure, yet subsequent unassisted creativity may suffer, with effects varying across divergent and convergent tasks and LLM strategies potentially narrowing idea diversity. The discussion stresses interaction design and evaluation beyond assisted performance, while noting limitations in task scope and exposure.
5.1.1 LLMs Boost Performance During Exposure, but Unassisted Participants Excel in Test.
- LLM assistance helped during exposure, but participants without prior LLM exposure generally performed better unassisted, including on divergent originality and convergent correct-word identification; not every difference was significant.Source: §5.1.1 · pp. 12–13
5.1.2 Differential Impact of LLMs on Divergent and Convergent Thinking.
- Effects differed by task: participants were less inclined to adopt LLM suggestions in divergent thinking, whereas structured convergent tasks appeared to benefit more during exposure, though AI may disrupt exploration or hasten solutions.Source: §5.1.2 · p. 13
5.1.3 Persistent Homogenization of Ideas and the Challenges of Designing LLM Coaches.
- LLM strategies reduced idea-set diversity during and after use, unlike direct ideas; participants without LLM interactions generated the most diverse sets, and convergent-test coach users underperformed direct-answer users.Source: §5.1.3 · p. 13
5.2 Broader Implications for Fields Involving Human-AI Co-Creativity
- Across science and the arts, evaluate how interaction design and post-use performance affect creativity: model capability alone cannot establish value, and systems should preserve long-term cognitive diversity.Source: §5.2 · pp. 13–14
5.3 Limitations & Future Work
- Short verbal AUT and RAT tasks, brief lab exposure, and static LLM interactions limit generalizability and leave long-term, nonverbal, and unconscious creativity insufficiently examined.AUT was limited to five objects to enable large-scale automated scoring. Future studies could use naturalistic and visual tasks and examine prolonged, adaptive LLM interactions.Source: §5.3 · p. 14
- Guidance was more verbose than direct answers and may have increased cognitive load; fatigue or increased dependency could also explain poorer coach-condition test performance, warranting controlled follow-up.Future studies should standardize guidance verbosity and investigate these mechanisms; the study’s static interactions may not reflect real-world conversational use.Source: §5.3 · p. 14
6 Conclusion
The experiments examine how repeated LLM assistance relates to human creativity across divergent and convergent thinking, suggesting that assistance can augment creativity while its mode may shape long-term creative abilities.
6 Conclusion
- The study reports a complex relationship between human creativity and LLM assistance, with the mode of assistance mattering for how it may shape long-term creative abilities.Source: §6 · p. 14
- Two parallel experiments examined LLM effects on divergent and convergent thinking, two key components of creative thinking.Source: §6 · p. 14
- Repeated exposure was used to approximate real-world interactions with LLMs and investigate lasting impacts on human creative processes.Source: §6 · p. 14
- The authors offer the work as a template for experimentally evaluating generative AI’s impact on human cognition and creativity.Source: §6 · p. 14
A Experiment 1 (Divergent Thinking) … A.1.2 Input Prompt: "What are some creative uses for a [OBJECT]? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. List creative uses for a [OBJECT]." .
The supplied passages specify the model configuration and identify a “List of Strategies” condition for Experiment 1’s divergent-thinking task.
A.1.1 Model Specification. •
- The model specification identifies gpt-4o as the model version used in August 2024.Source: §A.1.1 · p. 16
- The model was run at temperature 0 with top-p 0.95.Source: §A.1.1 · p. 16
- The model’s maximum response length was 4000.Source: §A.1.1 · p. 16
- Frequency and presence penalties were both set to 0.Source: §A.1.1 · p. 16
A.1.2 Input Prompt: "What are some creative uses for a [OBJECT]? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. List creative uses for a [OBJECT]." .
- The prompt material identifies a “List of Strategies” condition in Figure 11.Source: §A.1.2 · p. 16
A.2 Sample LLM Responses
Table 1 presents sample LLM responses in Experiment 1.
A.2 Sample LLM Responses
- Table 1 shows sample LLM responses in Experiment 1.Source: §A.2 · p. 16
B Experiment 2 (Convergent Thinking) … B.1.2 Input Prompt.
Experiment 2’s configuration specifies GPT-4o settings and includes materials describing the convergent word-association task and LLM prompt conditions.
B.1.1 Model Specification.
- GPT-4o was used in August 2024 with temperature 0, a maximum response length of 800, top-p 0.95, and frequency penalty 0.Source: §B.1.1 · p. 16
- The task prompt asks for a fourth word that forms a common compound word or phrase with each of three stimulus words.Source: §B.1.2 · p. 16
B.1.2 Input Prompt.
- Figures identify system prompts for the “List of Strategies” and LLM Guidance conditions, which were not visible to participants.Source: Figure 12 · p. 16, Figure 11 · p. 16
B.2 Sample LLM Responses
This section presents sample LLM responses from the exposure rounds of the convergent thinking experiment.
B.2 Sample LLM Responses
- Table 2 presents sample LLM responses for the convergent thinking experiment’s exposure rounds.Source: Table 2 · p. 18