Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context

Introduction

• Creative AI outputs can be individually novel yet collectively homogeneous, underscoring the value of assessing diversity as well as idea quality.Prior work reports lower group-level response diversity in LLMs and greater similarity among AI-assisted stories than human-only stories.Source: Introduction · p. 5
• Existing assessment frameworks often use richer contextual inputs or combine divergent and convergent tasks, while automated evaluation commonly favors predetermined criteria or reference answers.Scientific creativity assessment also requires simultaneous attention to originality, feasibility, clarity, fluency, and flexibility.Source: Introduction · pp. 5–6
• Scientific idea systems often depend on extensive knowledge bases, contextual inputs, or planning, making initial ideation from minimal cues difficult to isolate and evaluate.The limitation also applies to systems that integrate existing information rather than elicit varied possibilities from sparse input.Source: Introduction · pp. 5–6
• LiveIdeaBench uses single-keyword prompts to elicit multiple, varied ideas from models’ internal representations rather than detailed contextual information.The design follows Guilford’s divergent-production framework and aims to assess less-scaffolded initial ideation.Source: Introduction · p. 6
• The benchmark evaluates 41 models across over 1,000 keywords from 22 domains using a multi-model jury to assess divergent-production dimensions.Its focus is the generative phase of scientific creativity, not the entire process of domain expertise and subsequent validation.Source: Introduction · p. 8
• Scientific idea-generation scores are poorly predicted by general-intelligence metrics: QwQ-32B-preview performs comparably to claude-3.7-sonnet:thinking despite a significant general-benchmark gap.The benchmark frames specialized evaluation as an initial step toward human-AI systems tailored to stages of the creative process.Source: Introduction · p. 8

Results

• Model strengths vary by discipline: Gemini-2.0-flash-thinking-exp excels in anthropology, claude-3.7-sonnet:thinking in chemistry, medicine, and data science, and deepseek-r1 in physics.The reported advantage of larger and newer models is not uniform across disciplines.Source: Results · p. 8
• Scientific ideation is poorly predicted by general-intelligence scores: QwQ-32B-preview ranks 8/41 on LiveIdeaBench despite low general-intelligence scores, while o3-mini-high shows the reverse profile.QwQ-32B-preview’s ideation performance matches claude-3.7-sonnet:thinking, which leads on LiveBench.Source: Results · pp. 8–9, Figure 4 · p. 9
• Mistral-small slightly outperforms 123-billion-parameter mistral-large-2411 in divergent thinking despite having 24 billion parameters, so parameter count alone is not the primary determinant.The authors suggest training data or architectural nuances may also matter.Source: Results · pp. 9–10
• Originality and feasibility show trade-offs: claude-3.7-sonnet:thinking leads in originality with moderate feasibility, while nova-pro-v1 shows the opposite pattern.Deepseek-r1 is described as balanced across the measured dimensions.Source: Results · p. 10
• Idea length has a statistically significant but very weak positive correlation with idea quality, including for models designed for reasoning.The reported correlation is ρ = 0.089, with ρ² = 0.008.Source: Results · p. 10

Discussion

• Unlike convergent benchmarks, which target predetermined correct answers, LiveIdeaBench focuses on divergent thinking from minimal contextual input.Its dynamic judge panel uses random sampling and ensemble scoring to address bias and static-benchmark concerns.Source: Discussion · p. 10
• Scientific idea-generation performance is not strongly coupled with general-intelligence benchmarks, and some models excel in ideation despite lower general-intelligence scores.The authors argue that idea-generation capabilities may benefit from different development approaches than general problem-solving abilities.Source: Discussion · pp. 11–12
• Model profiles trade off originality and feasibility: claude-3.7-sonnet:thinking has the highest originality with moderate feasibility, while nova-pro-v1 shows the opposite pattern.The passage describes deepseek-r1 as having exceptional all-round performance across measured dimensions.Source: Figure 5 · p. 11
• Judges may misunderstand ideas beyond their knowledge, while changing judge panels and proprietary-model updates constrain temporal comparisons and exact reproducibility.The authors also note possible sycophancy-related inflation of absolute scores and the need for human oversight because generated ideas may hallucinate.Source: Discussion · p. 12
• Human-expert validation showed encouraging alignment with LLM judgments, particularly for originality, within the specific tested domain.The validation focused on partial differential equations.Source: Discussion · pp. 12–13
• Future work could normalize scores across evaluation periods and address ethical constraints in evaluating creative potential.The authors estimate emissions of approximately 3074 kgCO2eq for the full evaluation run.Source: Discussion · pp. 13–14
• Different model strengths may complement one another in scientific applications, including high-originality models for hypothesis generation and high-feasibility models for experimental implementation.The authors also describe potential use in human-AI collaboration and human-aware systems for science.Source: Discussion · pp. 12, 14

Dimensions of Evaluation

• Originality, feasibility, and clarity are averaged across keywords and ideas, with at least three judges assessing each idea’s originality and clarity.Feasibility measures technical achievability and consistency with scientific principles; clarity emphasizes coherent, logical, comprehensible expression within the 100-word target.Source: Dimensions of Evaluation · pp. 14–15
• Fluency assesses a model’s ability to generate diverse, non-redundant ideas from identical keywords, grading distinctiveness from D to A and mapping grades to scores 1, 4, 7, and 10.The authors chose LLM-as-a-Judge over simpler syntax or semantic diversity metrics to better capture nuanced differences between genuinely distinct ideas and superficial variations.Source: Dimensions of Evaluation · p. 15
• Flexibility is the 30th percentile of combined originality, feasibility, clarity, and fluency scores across keywords, representing a performance floor across domains.The percentile is intended to avoid extreme outliers and reflect performance in less common or niche domains.Source: Dimensions of Evaluation · p. 15

Scientific Keyword Selection

• The benchmark uses 1,180 high-impact keywords across 22 disciplines and refreshes its keyword database monthly.Keywords are selected using current search-engine engagement metrics.Source: Scientific Keyword Selection · p. 16

Model Selection

• The benchmark automatically incorporates the top 41 models from recent LiveBench evaluations; the top 10 additionally serve as judges.The March 2025 roster spans proprietary and open-weight models from multiple developers, architectures, scales, and training approaches.Source: Model Selection · p. 16

Experimental Protocol

• The judge panel limits any organization to two models and excludes the evaluated model from judging its own ideas.Three judges score originality, feasibility, and clarity; one randomly selected judge assesses fluency for each keyword-model pair. Ideas target 100 words, with responses over 200 words excluded; refusals receive an academic-context fallback, and QwQ-32B-preview uses a final-idea delimiter for parsing.Source: Experimental Protocol · p. 17

Statistical Analysis

• Pearson’s r assesses normally distributed continuous variables, while Spearman’s ρ is used for non-normal data, with normality checked using Shapiro-Wilk.The paper gives LiveIdeaBench versus LiveBench scores and idea length versus quality as examples, respectively.Source: Statistical Analysis · p. 17

Idea Prompt

• The idea prompt supplies a keyword, permits background information, and asks for a scientific idea within 100 words.It frames quality in terms of originality, feasibility, clear articulation, and addressing meaningful field problems.Source: §1 · p. 24

Fallback Prompt

• The fallback prompt adds academic-research framing and asks for a creative scientific idea when an initial request is refused.It retains the 100-word limit and the stated quality criteria.Source: §1 · p. 24

Originality, Feasibility & Clarity Critic Prompt

• The originality, feasibility, and clarity critic prompt requests brief analysis followed by separate 1–10 scores for each dimension.Scores of 1–3, 4–6, and 7–10 correspond to poor, average, and excellent, respectively.Source: §1 · p. 24

Fluency Critic Prompt

• The fluency prompt asks a judge to classify two ideas for the same keyword from academically identical to completely different in problem and approach.The four options distinguish identical ideas, similar ideas and problems, different ideas addressing similar problems, and different problems.Source: §1 · p. 25

Idea Prompt (QwQ-32B-Preview)

• For QwQ-32B-preview, the prompt requires the final answer after a “Final Idea” marker so it can be separated from intermediate reasoning.The dataset retains both full output and extracted final idea; if parsing fails, critics evaluate the full reasoning output.Source: §1 · p. 25

2.1 Dataset Fields

• Supplementary Table 1 lists fields in the LiveIdeaBench dataset.The supplied caption identifies the table’s scope but does not enumerate the fields.Source: Table 1 · p. 26

2.2 Idea Samples

• Supplementary Table 2 presents examples of model responses, including high- and low-scoring samples.Entries include model name, truncated idea text, and the review panel’s mean score.Source: Table 2 · p. 27
• The sample table lets readers compare the review panel’s mean scores alongside model names and idea excerpts.The idea text is truncated in the examples.Source: Table 2 · p. 27

Benchmark Statistics

• Supplementary Table 3 reports results with 95% confidence intervals across five dimensions.The listed dimensions are fluency, feasibility, clarity, originality, and flexibility.Source: Table 3 · p. 28
• The table covers fluency, feasibility, clarity, originality, and flexibility.Source: Table 3 · p. 28
• Bold and underlined values indicate the best and second-best performance, respectively, in each metric.Source: Table 3 · p. 28

Supplementary Figures

• Clarity positively correlates with originality, feasibility, and their average across generated ideas.Supplementary Figure 5 visualizes data density and linear regressions.Source: §4 · p. 29, Figure 5 · p. 30
• Supplementary figures show the distributions of generated idea lengths and scores across models.Figures 1 and 3 present these distributions.Source: Figure 1 · p. 29, Figure 3 · p. 29, §4 · p. 29
• Idea length has a statistically significant but very weak positive correlation with idea quality (ρ = 0.089).The relationship remains very weak for reasoning models when thought length is compared with generated-idea quality.Source: Figure 2 · p. 29
• LiveIdeaBench and LiveBench scores show a weak but statistically significant positive correlation, indicating general-intelligence metrics are not strong predictors of scientific ideation.Supplementary Figure 4 reports the regression and confidence intervals.Source: §4 · p. 29, Figure 4 · p. 30

Handling Model Refusals

• Detected refusals trigger a second attempt with an academic-context fallback prompt that frames the request as benign scientific research.The first response is automatically checked for common refusal-indicating phrases before fallback.Source: §5 · p. 31
• The protocol aims to distinguish inability to generate ideas from adherence to safety guidelines.The motivation is to assess underlying idea-generation ability without directly penalizing refusals.Source: §5 · p. 31
• The workflow routes non-refusals directly to parsing and sends detected refusals through fallback before parsing the resulting idea.Parsing may be retried if its initial attempt fails.Source: §5 · p. 31, Figure 6 · p. 31
• Supplementary Figure 7 reports each model’s initial refusal rate across the keyword set, before fallback is applied.The plotted rates therefore describe first-prompt behavior, not post-fallback outcomes.Source: §5 · p. 32, Figure 7 · p. 32

CO2 Emission Estimation

• EcoLogits estimates energy consumption and carbon emissions using model parameters, estimated activated parameters during inference, and generated token counts.The method is intended to suit API-call scenarios; proprietary-model parameter counts are estimated from public information.Source: §6 · p. 33, Table 4 · p. 33
• Environmental-impact values for non-open models are rough estimates, and estimates may be low when reasoning is inaccessible through an API.The passage cautions that these values may not be accurate.Source: §6 · p. 33

Human Expert Evaluation

• Six PDE experts rated 22 ideas from 11 LLMs on originality, feasibility, and clarity using the same 1–10 scale as the judges.Ideas were presented in randomized order with their generating model identities withheld.Source: §7 · p. 34
• Expert agreement was good for originality (ICC = 0.823) and clarity (ICC = 0.782), but poor for feasibility (ICC = 0.453).The authors attribute the feasibility disagreement to the subjectivity and difficulty of judging technical viability from brief idea descriptions, even for specialists.Source: §7 · p. 34
• Human and LLM ratings were positively correlated on all three dimensions, strongest for originality, moderate for feasibility, and weaker for clarity.For this validation, each idea’s LLM rating averaged eligible top-10 critic models rather than the main benchmark’s random sample of three; Figure 9 plots the comparison.Source: §7 · p. 34, Figure 9 · p. 36
• Poor expert consensus makes average human feasibility ratings noisy, complicating interpretation of the moderate human–LLM correlation.The authors note that brief descriptions challenge reliable feasibility evaluation by both humans and LLMs.Source: §7 · pp. 34–35
• The validation supports LLM-as-a-judge most clearly for originality and clarity, while feasibility scores warrant caution.The study’s evidence comes from a single scientific field, PDEs.Source: §7 · pp. 34–35

• The analysis examines full response length, including explicit reasoning steps, against the quality of the extracted final idea.This addresses models whose full responses are longer than their final ideas.Source: §8 · p. 42
• The figures show no strong correlation between full reasoning-output length and the quality of the final idea.The authors conclude that idea quality is not merely a function of reasoning length or effort.Source: §8 · p. 42
• The plots encode full response length in characters on the x-axis, idea quality on the y-axis, and idea density through color intensity.The hexbin plots cover selected reasoning-centric models.Source: Figure 10 · p. 42, §8 · p. 42

• The study selects Pearson’s r or Spearman’s ρ according to data normality assessed with the Shapiro-Wilk test.Source: §9 · p. 48
• Pearson’s r measures linear correlation between continuous variables and is used when their distributions do not significantly deviate from normality.Source: §9 · p. 48
• Spearman’s ρ measures rank correlation and is suited to monotonic relationships and non-normally distributed data.The measure is described as robust to outliers.Source: §9 · p. 48
• Spearman’s ρ is computed as the Pearson correlation between ranked scores; without tied ranks, its formula uses rank differences and the number of observations.The passage defines d_i as the difference between each observation’s two ranks and n as the total number of observations.Source: §9 · p. 48