LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play

1 Introduction

• Prior LLM collaboration has mostly targeted closed-ended tasks, while LLM creativity research has mostly used single-agent setups.Source: §1 · p. 1
• Commercial LLMs are not specifically trained for multi-turn conversations, and their homogeneity can hinder divergent discussion and lead to quick convergence.Source: §1 · p. 2
• The framework uses specialized prompts across three phases to encourage vigorous, divergent exchanges and convergence to creative answers.Role-play assigns participating LLMs diverse backgrounds and perspectives, including roles such as Visionary Millionaire and Futurist.Source: §1 · p. 2
• Across four creativity benchmarks, LLM Discussion outperforms baselines in producing creative answers, evaluated by LLMs and humans using four metrics.The metrics are Originality, Elaboration, Fluency, and Flexibility.Source: §1 · p. 2

• Prior multi-LLM frameworks target factuality, reasoning, arithmetic, and coding, mostly evaluating on MMLU, GSM8K, and HumanEval.This work instead explores LLM creativity through a multi-LLM discussion framework.Source: §2 · p. 2
• Studies of LLM creativity have covered creative writing, metaphor assessment, and alternative object uses, but are mainly single-LLM setups.This work instead employs multiple LLMs to induce collective creativity.Source: §2 · p. 2
• Prior work finds that role-playing can induce human-like behavior and interactivity and help LLMs tackle complex tasks.This work harnesses role-playing to enhance LLM creativity.Source: §2 · p. 2

3 Approach: LLM Discussion

• LLM Discussion combines a three-phase framework with role-play so multiple agents can build on one another’s thoughts and produce creative outcomes.Its short-term phase objectives and specialized prompts support both divergent and convergent thinking.Source: §3 · p. 3

3.1 Discussion Framework

• Unlike LLM Debate (Du et al., 2023), which maintains agents’ own answers while they observe others, the framework explicitly prompts agents to build on peers’ responses.The paper notes that debate can resolve factual questions but may fall short of encouraging divergent thinking that builds on others’ ideas.Source: §3.1 · p. 3
• Initiation establishes the task and group context, discussion rounds provide peers’ previous responses for idea exchange, and convergence prompts a final list of creative answers.The phases respectively set up the task, encourage agents to generate and follow up on ideas, and summarize discussion ideas into a collective conclusion.Source: §3.1 · p. 3
• The procedure starts with initiation, repeats discussion prompts with other agents’ responses, and uses a final convergence prompt to reach a collective conclusion.Source: §3.1 · p. 4

3.2 Role-Play

• Role-play addresses agent homogeneity by assigning diverse backgrounds, perspectives, and experiences; the paper also draws on ideation roles rather than problem-oriented roles.Examples include Six Thinking Hats perspectives such as emotion and innovation.Source: §3.2 · p. 4
• The role set is an initial attempt at automated generation; systematically exploring alternative role-generation mechanisms remains future work.Source: §3.2 · p. 5
• Agents receive distinct roles before initiation, are reminded of their roles and responsibilities each round, and keep the same roles throughout the discussion.Roles are selected autoregressively, and speakers declare their roles to the other participants.Source: §3.2 · p. 5

4.1 Benchmarks

• Figure 3 depicts each LLM receiving a role with specialties and a detailed role prompt at discussion start.Source: Figure 3 · p. 5
• Across AUT, INSTANCES, SIMILARITIES, and SCIENTIFIC, the benchmarks assess everyday creative thinking and scientific creativity or process skills.Their tasks include practical object uses, listing items by criteria, finding commonalities, and solving scientific problems.Source: §4.1 · p. 5

4.3 Evaluation

• The evaluation uses four established creativity benchmarks and quantitative ratings from both LLMs and humans.Source: §4.1 · p. 5, §4.3 · p. 6

4.2 Baseline

• The baselines include a zero-shot single-agent prompt, Brainstorm then Select for AUT, and multi-LLM Debate.Zero-shot was selected as the main single-agent baseline because no tested prompt consistently outperformed the others.Source: §4.2 · pp. 5–6

4.4.1 LLM Evaluation Results

• LLM Discussion outperforms Single Agent, Brainstorm, then select, and LLM Debate in Originality on four benchmarks and Elaboration on three.Table 2 reports the LLM evaluation results across AUT, INSTANCES, SIMILARITIES, and SCIENTIFIC.Source: §4.4.1 · p. 7, Table 2 · p. 8
• In Figure 4, responses and LLM Originality scores for AUT and SCIENTIFIC show more innovative, detailed ideas from LLM Discussion.Examples include role-specific suggestions and more developed concepts than baseline responses.Source: §4.4.1 · pp. 7–8, Figure 4 · p. 8

4.4.2 Human Evaluation and Correlation

• Human evaluation finds LLM Discussion outperforms baselines in Originality and Elaboration on three of four benchmarks.The result aligns with the LLM evaluation.Source: §4.4.2 · p. 8, Table 3 · p. 9
• LLM evaluations correlate strongly with average human evaluations for Originality, Fluency, and Flexibility, and moderately for Elaboration.LLM-to-average-human correlations exceed correlations between individual human ratings and the average, which show high variance.Source: Table 4 · p. 9, §4.4.2 · p. 9

5 LLM Discussion

• LLM Discussion uses three specialized discussion phases and diverse roles to encourage creative exchange and convergence.The roles give LLMs distinct backgrounds, perspectives, and personalities.Source: §5 · p. 10
• Creativity is difficult to quantify, so expert interviews could provide more comprehensive evaluation.The authors also identify human–LLM interaction as a direction for studying collective creativity.Source: §5 · p. 10

A Baseline Result

• The baseline comparison includes single-LLM prompting variants and multi-LLM frameworks, with results collected in Table 5.Single-LLM variants include zero-shot, few-shot, zero-shot-CoT, LLM stimuli, and “Take a deep breath”; multi-LLM baselines include Brainstorm, then Select on AUT and LLM Debate across datasets.Source: A Baseline Result · p. 15, Table 5 · p. 15

B Role-Play Role Sets

• Tables 6–8 document role-play and Six Thinking Hats settings, plus single-agent results using both role sets.The settings include roles, specialties, and prompts generated by GPT-4.Source: B Role-Play Role Sets · p. 16, Table 6 · p. 16, Table 7 · p. 17

C Evaluation Prompts

• Table 9 contains the evaluation prompts for the four creativity metrics.Source: C Evaluation Prompts · p. 17

D Ablation Study for Prompts

• Prompt 1 was selected for the main experiments because it had the most stable performance across four datasets.The initiation-prompt ablation compares GPT-4-generated candidates, with results reported in Table 11.Source: D Ablation Study for Prompts · p. 17

METRICS PROMPT

• Originality is scored from 1 to 5 according to response novelty, ranging from very common to extremely unique and rare.The rubric asks evaluators to provide a brief justification with examples.Source: METRICS PROMPT · p. 18
• Table 9 presents prompts for evaluating the four creativity metrics.Source: Table 9 · p. 18

PROMPT CONTENT

• Prompt 1 asks participants to answer divergently and creatively as members of a group discussion.Source: PROMPT CONTENT · p. 19
• Prompts 2 and 3 encourage idea-building, unexpected connections, and unconventional perspectives.Prompt 2 frames the task as brainstorming; Prompt 3 frames it as a think tank.Source: PROMPT CONTENT · p. 19
• Prompt 4 asks participants to contribute distinct insights, challenge ideas, and expand contributions through feedback and questioning.Its stated objective is a broad spectrum of ideas and solutions.Source: PROMPT CONTENT · p. 19
• Prompt 5 frames creative problem-solving as a group mission to piece together clues from members’ ideas.Source: PROMPT CONTENT · p. 19
• Table 10 lists GPT-4-generated initiation prompts intended to improve divergent thinking, and Table 11 reports their evaluation on four benchmarks.Source: Table 11 · p. 19, Table 10

E Correlation Between Length and Elaboration

• Humans tend to give higher elaboration scores to longer responses, with response length strongly correlated with those scores.The analysis followed a divergence in Kendall’s τ between LLM and human elaboration evaluations.Source: E Correlation Between Length and… · pp. 19–20, Table 12 · p. 20

F Analysis of Response Length and Its Impact on Scores

• Table 13 reports average response word counts and standard deviations for single agent, LLM Debate, and LLM Discussion across four benchmarks.The supplied passage gives the table’s scope but no cell values.Source: F Analysis of Response Length and Its… · p. 20, Table 13 · p. 20
• Originality correlations with response length are below 0.3, while Elaboration is moderately correlated across frameworks and benchmarks.Table 14 reports Pearson correlations between word count and creativity scores.Source: F Analysis of Response Length and Its… · p. 20, Table 14 · p. 20
• Within the same word-length intervals, LLM Discussion outperforms other baselines in Originality and Elaboration on most benchmarks.Figures 7 and 8 present the comparisons by response-length interval.Source: F Analysis of Response Length and Its… · p. 20

G Why are Fluency and Flexibility Less Important

• Originality and Elaboration are prioritized as metrics that more directly reflect creative potential, while Fluency and Flexibility are not prioritized.The authors cite possible score inflation from LLMs’ ability to generate many responses.Source: G Why are Fluency and Flexibility Less… · p. 20
• Guzik et al. (2023) and Chi (2024) link high Fluency and Flexibility scores to rapid response generation, while Hadas & Hershkovitz (2024) report that Flexibility scores are significantly affected by Fluency scores.These cited works inform the authors’ rationale for deprioritizing the two metrics.Source: G Why are Fluency and Flexibility Less… · p. 20
• As many as possible prompting during convergence easily increases Fluency and Flexibility scores.Table 15 compares LLM Discussion with the AMAP prompt.Source: G Why are Fluency and Flexibility Less… · p. 22, Table 15 · p. 22

H Analysis of Temperature and Its Impact

• Increasing temperature from 0.0 to 2.0 raises diversity, as shown by lower self-BLEU, but does not yield higher creativity scores on AUT.Temperature 1.0 was used in the main experiments; Figure 9 shows minimal creative-score variation as temperature increases.Source: H Analysis of Temperature and Its Impact · p. 22, Figure 9 · p. 22
• At temperatures of 1.6 or higher, model outputs include nonsensical responses such as unrecognizable symbols.The temperature experiment was conducted on AUT.Source: H Analysis of Temperature and Its Impact · p. 22
• Higher temperatures seemed to reduce Originality and Elaboration in some cases despite increasing diversity.The passage states that diversity did not correlate with higher creativity.Source: H Analysis of Temperature and Its Impact · p. 22

I Analysis on Three-Phase Discussion Framework and Role-Play

• Table 16 compares role-play without three-phase discussion, three-phase discussion without role-play, and the full LLM Discussion framework across four benchmarks.The supplied passages identify the ablation setup but do not provide its result values.Source: I Analysis on Three-Phase Discussion… · p. 22, Table 16 · p. 23

J Role Play Generation

• The authors use role storming to encourage agents to approach creative tasks from different perspectives, and say their GPT-4-generated role list is not optimal.They identify broader demographic coverage as important future work.Source: J Role Play Generation · p. 23

Round 1 - User Prompt

• The Round 1 prompt asks for personas to diversify ideation, and the response proposes roles including a Visionary Millionaire, Startup Founder, Creative Professional, and Futurist.The response describes these personas as offering different perspectives for creative thinking.Source: Round 1 - User Prompt · pp. 23–24

Round 3 - User Prompt

• One Round 3 prompt requests JSON role specifications, and the response includes persona fields such as agent name, role, specialty, and role prompt.Examples include the Visionary Millionaire and Startup Founder.Source: Round 3 - User Prompt · pp. 24–25
• A separate Round 3 exchange specifies White Hat and Red Hat roles, with prompts focused on information analysis and emotional responses.The response presents the two roles in JSON-like form.Source: Round 3 - User Prompt · p. 24

K Chat Log

• LLM Discussion shows collaborative exchange and combines sustainability, creativity, technology, and human engagement, while LLM Debate mainly updates answers iteratively without discussion.The paper describes the debate approach as less likely to produce original creative solutions.Source: K Chat Log · pp. 26–27

Round 1 - Agent Response

• The environmentalist role opens with ten umbrella ideas spanning wildlife shelter, plant holders, art, storage, lighting, and play.The prompt asks the agent to brainstorm creatively and diversely while considering sustainability and environmental health.Source: Round 1 - Agent Response · p. 27, Round 1 - User Prompt · p. 27

Round 2 - Agent Response

• The next round expands the brainstorm with practical and aesthetic repurposing ideas, including a bike rack, birdhouse, plant stand, and fashion accessory.The environmentalist response follows other agents’ creative-professional and futurist suggestions.Source: Round 2 - User Prompt · p. 28, Round 2 - Agent Response · p. 29

Round 3 - Agent Response

• Round 3 adds community and event concepts, such as an outdoor cinema, scavenger hunt, storytelling circle, language exchange, and sustainability exhibition.The environmentalist builds on teammates’ suggestions, which also include technology-oriented umbrella concepts.Source: Round 3 - User Prompt · p. 30, Round 3 - Agent Response · p. 31

Round 4 - Agent Response

• Round 4 continues with artistic, wellness, and sustainable uses, from upcycled garden sculptures and community murals to solar-panel picnic tables.The response also imagines umbrellas as sound sanctuaries, VR controllers, and biophilic office decor.Source: Round 4 - Agent Response · pp. 33–34

Round 5 - User Prompt

• The Round 5 prompt continues the exchange with further proposals, including umbrella art therapy, networking, cultural exchange, and upcycling workshops.The supplied passage ends partway through the list of suggestions.Source: Round 5 - User Prompt · p. 34
• The final-round user prompt asks the group to finalize a numbered list of as many diverse, creative umbrella uses as possible.It defines creative ideas as clever, unusual, interesting, uncommon, humorous, innovative, or different.Source: Round 5 - User Prompt · p. 36

Round 5 - Agent Response

• The Round 5 agent response suggests umbrella activities for therapy, networking, cultural exchange, upcycling, dance, storytelling, sound healing, and mindfulness.It also proposes an innovation symposium, self-expression parade, and outdoor retreat.Source: Round 5 - Agent Response · p. 36

Agent 1 Environmentalist - Initiation Prompt

• The environmentalist initiation prompt frames the task around eco-friendly ideas that support sustainability and environmental health.It asks the agent to claim this role and initiate a group discussion.Source: Agent 1 Environmentalist - Initiation… · p. 37

Agent 2 Creative Professional - Initiation Prompt

• The creative-professional initiation prompt emphasizes aesthetics, narrative, emotion, and expressive ideas that resonate with people.It asks the agent to claim this role and initiate a group discussion.Source: Agent 2 Creative Professional -… · p. 37

Agent 3 Futurist - Initiation Prompt

• The futurist initiation prompt directs the agent to explore emerging technologies and future scenarios while proposing diverse, creative umbrella uses.It asks the agent to claim this role and initiate a group discussion.Source: Agent 3 Futurist - Initiation Prompt · p. 37

Agent 1 Environmentalist - Final Response

• The environmentalist’s final list pairs community events and wellness activities with umbrella art, cultural exchange, and upcycling.Suggestions include networking, dance, storybook reading, sound healing, a symposium, a parade, and a mindfulness retreat.Source: Agent 1 Environmentalist - Final… · p. 38

Agent 2 Creative Professional - Final Response

• The creative professional proposes umbrella-based art, sound, fashion, virtual reality, urban farming, marine conservation, and public data displays.The ideas span artistic expression, technology, sustainability, and community engagement.Source: Agent 2 Creative Professional - Final… · pp. 38–39

• Umbrella concepts span nature-inspired experiences, recycled-material fashion and sculpture, virtual reality, botanical art, and solar-powered outdoor design.Source: Agent 3 Futurist - Final Response · p. 39
• Other proposals use umbrellas for community art, biophilic workplaces, wellness activities, and marine-conservation outreach.Source: Agent 3 Futurist - Final Response · p. 39
• The response adds a bio-printing studio and neon-lit night market as art-science and retail concepts.Source: Agent 3 Futurist - Final Response · pp. 39–40
• Further ideas include interactive play, mindfulness, renewable-energy and technology showcases, sensory gardens, disaster response, and cultural exchange.Source: Agent 3 Futurist - Final Response · p. 40