Large Language Models for Scientific Idea Generation: A Creativity-Centered Survey

1 Introduction

• Scientific ideation requires novelty and valueness jointly; novelty without value can be arbitrary, while value without novelty can be incremental or redundant.Valueness includes correctness, plausibility, feasibility, or usefulness in a scientific context.Source: §1 · p. 2
• Boden’s taxonomy characterizes expected novelty as combinatorial, exploratory, or transformational, while Rhodes’ 4Ps identify creativity’s person, process, press, and product dimensions.The assignments are conceptual tendencies, not guaranteed outcomes.Source: §1 · p. 3
• The survey groups approaches into knowledge augmentation, prompt steering, inference-time search, multi-agent systems, and parameter adaptation.These families respectively emphasize external knowledge, input manipulation, iterative or branching exploration, collaboration, and model-parameter changes.Source: §1 · pp. 3–4
• Evaluation practices remain challenged by subjectivity, reliability, and comparability, placing the assessment of generated ideas within the product dimension of Rhodes’ 4Ps.The survey treats evaluation as a central part of understanding scientific creativity.Source: §1 · p. 4
• Transformational creativity remains elusive, while person- and product-focused perspectives are underexplored and standardized evaluation metrics and benchmarks are lacking.The survey also notes possible structural limits from autoregressive training and attention-based architectures.Source: §1 · p. 4

2 Knowledge Augmentation: Grounding LLMs in External Evidence

• Knowledge augmentation supplies external literature and factual resources to ground generation, reduce hallucinations, and support combinatorial creativity through recombination.The survey distinguishes semantic retrieval from structured relational grounding and links external context to Rhodes’ press dimension.Source: §2 · p. 6

2.1 Semantic Retrieval

• Semantic retrieval injects papers or text fragments selected by similarity into the prompt; systems range from raw-literature access to multi-stage, citation-aware frameworks.PaperQA (Lála et al., 2023) and LitLLM (Agarwal et al., 2024) are early examples; newer systems include Ideasynth (Pu et al., 2025) and ResearchAgent (Baek et al., 2024).Source: §2.1 · p. 6
• Similarity-driven retrieval can keep exploration close to known knowledge clusters, limiting creativity.The survey describes this approach as effective for relevance but potentially restrictive for broader exploration.Source: §2.1 · p. 6

2.2 Relational Retrieval

• Knowledge graphs encode relations among concepts, methods, results, or papers, enabling cross-domain paths and multi-hop reasoning beyond local similarity.The survey describes graph-based methods as supporting more multidisciplinary and potentially more creative hypothesis generation.Source: §2.2 · pp. 6–7

2.3 Takeaways and Limitations

• GoAI scores 3.83 on Elo-based novelty and 3.39 on overall average, versus Chain-of-Ideas at 3.21 and 3.27, and vanilla prompting at 2.44 and 2.70, respectively.The survey interprets the result as evidence that relational structuring supports combinatorial creativity while maintaining feasibility and effectiveness.Source: §2.3 · p. 7
• Heavy reliance on retrieved content can bias ideas toward well-represented clusters, reduce diversity, and encourage conservative or incremental thinking.The survey identifies balancing grounding with exploration and explicitly encouraging divergence as directions for addressing these limitations.Source: §2.3 · p. 7

3 Prompt-Driven Creativity: Steering LLMs Towards Novel Ideas

• Prompt engineering steers generation through instructions, roles, or structured templates without costly retraining or fine-tuning.The survey frames prompts as external constraints and affordances associated with the press dimension of the 4Ps.Source: §3 · p. 10

3.1 Persona & Role Priming

• Role prompts such as “act as a scientist” can boost originality on psychometric creativity tests, while persona design also raises consistency and bias considerations.The cited studies also report that ambiguous or counterintuitive personas can spur unexpected ideas.Source: §3.1 · p. 10

3.2 Constraint-Based & Adversarial Queries

• Constraint-based and adversarial prompts push models toward less obvious solutions by denying routine strategies and exploring lower-probability outputs.Denial Prompting is evaluated with the NeoGauge creativity metric (Lu et al., 2024b).Source: §3.2 · p. 10

3.3 Structured Creative Prompts

• Structured prompts scaffold reasoning: Chain-of-Thought encourages stepwise thinking, while Consistency-Based Self-Adaptive Prompting uses model predictions and unlabeled data to construct prompts.The cited work reports improved reasoning accuracy and idea diversity for Chain-of-Thought (Wei et al., 2022).Source: §3.3 · pp. 10–11
• Chain-of-Ideas organizes literature into sequential knowledge chains, while Bit-Flip-Spark moves from a baseline assumption to a challenge and then a novel insight.These structures aim to trace research development or question default assumptions.Source: §3.3 · p. 11
• Structured scaffolds may approximate aspects of transformational behavior through controlled search and assumption relaxation, rather than true representational change.The survey links these scaffolds to reweighting exploration within a fixed pretrained space.Source: §3.3 · p. 11

3.4 Multilingual Prompting

• Multilingual prompting can increase diversity and creativity by drawing on distinct linguistic and cultural priors; a survey spanning more than 250 languages reports broad effectiveness.The cited findings also associate cross-lingual reformulations with improved comprehension and reasoning.Source: §3.4 · p. 12

3.5 Takeaways and Limitations

• Instructional prompting raises novelty from 3.78 to 4.05 and valueness from 4.46 to 4.65; Chain-of-Thought primarily raises valueness, while aligned models may resist strong creativity cues.The survey says prompting redistributes generation within existing knowledge with limited impact on deeper exploratory novelty; alignment training is described as prioritizing safety and helpfulness over novelty.Source: §3.5 · p. 12

4 Search and sampling expansions: Inference-Time Exploration of Hypothesis Spaces

• Search mechanism, feedback source, and idea abstraction jointly shape how inference-time methods navigate novelty and feasibility.Search ranges from local refinement to tree or population search, feedback from self-checks to experts, and abstraction from tokens to hypotheses.Source: §4 · p. 13

4.1.1 Local Search: Single-Candidate Refinement

• Single-candidate refinement is simple and low-cost but can follow a local maximum and miss a solution entirely.Sequential refinement improves one hypothesis through feedback cycles.Source: §4.1.1 · pp. 13–14

4.1.2 Population-Based Search: Beam-Style Refinement

• Population-based search maintains several candidates to balance exploration and exploitation, reducing local-optimum risk, but discarded branches cannot be recovered.Multiple hypotheses are propagated across iterations, yet initially underperforming directions may be lost.Source: §4.1.2 · p. 14

4.1.3 Tree Search: Branching Exploration

• Tree search expands, revisits, and backtracks across paths, offering broader exploration than beam search but at significant computational cost.Preserving unconventional hypotheses depends on the search heuristics.Source: §4.1.3 · pp. 14, 16

4.2.1 Internal Self-Evaluation

• Internal confidence supports self-critique without external input, but works best when pretrained linguistic knowledge suffices and can be miscalibrated.Self-Refine, PANEL, and CriticAL use model-generated critiques or statistical reasoning to identify weaknesses.Source: §4.2.1 · p. 16

4.2.2 Peer LLMs as Verifiers

• Peer evaluators can offer domain specialization and rationales, but generators may exploit verifier weaknesses to earn high ratings without genuine quality.Their reliability depends on careful design that limits reward hacking and preserves evaluation diversity.Source: §4.2.2 · p. 18

4.2.3 Simulators and Real Environments; 4.2.4 External Tools and APIs

• Simulators return grounded responses for iterative hypothesis refinement, while external tools and APIs provide tool-call feedback that can improve decision-making.Lab-in-the-loop systems feed simulation or experimental outputs back into ideation; MINT studies tool use across multiple turns.Source: §4.2.3 · p. 18, §4.2.4 · p. 19

4.2.5 Scientific Priors and Domain Rules; 4.2.6 Human-in-the-Loop Evaluation

• Scientific priors reliably constrain ideas in formalized fields but can be rigid, while human evaluation suits subjective domains yet is costly and hard to scale.Human judgment can help where intuition and context matter, including when simulations or priors leave gaps.Source: §4.2.5 · p. 19, §4.2.6 · pp. 20–21

4.3.1 Hypothesis-Level Ideation

• At the hypothesis level, systems propose conceptual hypotheses, causal explanations, or theoretical insights for early-stage brainstorming, with downstream verification still required.MAGIC ranks speculative hypotheses using plausibility and novelty heuristics; VirSci uses debate, and IRIS uses reflection to refine proposals.Source: §4.3.1 · p. 21

4.3.2 Program-Level Ideation

• Program-level ideation produces executable protocols or code, integrating external tools and returned results into the generation loop.Execution can make outputs structured and interpretable and reduce unsupported claims through empirical validation.Source: §4.3.2 · pp. 21–22

4.3.3 Meta-Level Search and Self-Adaptation

• Meta-level search adapts the agent’s own configuration or environment interaction, including prompts, exploration parameters, reward weights, and architecture.The Darwin Gödel Machine evolves its reasoning code, while the AI co-scientist Meta-review agent uses accumulated feedback to adjust prompts.Source: §4.3.3 · p. 23
• Across abstraction levels, search ranges from hypotheses to executable plans to changing the rules that generate them.The survey presents meta-level flexibility as a possible contributor to scaling LLMs into autonomous scientific explorers.Source: §4.3.3 · p. 23

4.4 Takeaways and Limitations

• Monte Carlo Thought Search reaches rewards of 12.47–15.6, versus 9.91–13.8 for Tree-of-Thought and 2.04–2.27 for standard CoT, with deeper, adaptive exploration.The comparison reports NP ≈301 and depth ≈9 for Monte Carlo Thought Search, versus NP = 253 and depth = 5 for Tree-of-Thought; gains cost more computation and depend on evaluator quality.Source: §4.4 · pp. 23, 25

5.1 Pipeline-Oriented Workflows: Automation of Complex Workflows

• Pipeline-oriented systems assign specialized agents distinct research roles to automate stages such as literature review, hypothesis generation, experiment design, and writing.Robin computationally validated a novel therapeutic candidate for dry age-related macular degeneration; these pipelines primarily target efficiency and automation rather than novelty.Source: §5.1 · pp. 26–27

5.2 Debate and Emergent Collaboration: Fostering Creative Discovery

• Debate-oriented systems use multi-turn critique and revision, often with generators and critics, to challenge, defend, and synthesize hypotheses.VirSci uses brainstorming agents, critics, and judges; Multi-Agent Debate uses adversarial roles and a neutral referee to scrutinize proposals.Source: §5.2 · pp. 27–28
• Embedding debate in structured pipelines can make it both a filter for weak ideas and a driver of exploration toward diverse hypotheses.AI-Coscientist combines generation, reflection, ranking, and evolutionary search with debate.Source: §5.2 · p. 28

5.3 Takeaways and Limitations

• The survey distinguishes workflow automation from emergent creativity through agent interaction, identifying process as a source of creativity in the 4Ps framework.It describes specialized agents as dividing complex workflows and debate or critique as enabling novel exploratory hypotheses.Source: §5.3 · p. 28, Figure 11 · p. 30
• VirSci multi-agent systems outperformed single-agent baselines across backbones, with human ratings of novelty 3.78 vs. 3.10–3.22, feasibility 5.24 vs. 4.78–4.94, and effectiveness 4.95 vs. 4.43–4.77.The survey also reports an AI co-scientist rediscovery of an unpublished bacterial gene-transfer mechanism absent from public data, which it treats as an early signal of transformational creativity.Source: §5.3 · p. 28
• Multi-agent systems face brittle prompting, hand-tuned orchestration, cascading failures, and substantial computational overhead.Their advantages diminish as base LLMs improve, and strong single-agent models can often match them without communication costs.Source: §5.3 · pp. 28–29

6.1 Supervised Fine-Tuning Approaches

• Supervised fine-tuning uses curated scientific or instruction datasets to impart domain expertise, but can overfit or fail to generalize beyond its training distribution.The DARWIN series uses more than 20,000 instruction–answer pairs, while OmniScience combines domain-adaptive pretraining, instruction tuning, and reasoning-based distillation.Source: §6.1 · p. 31

6.2 Reinforcement Learning-Based Methods

• Reinforcement learning aligns models with goal-driven scientific tasks, with process reward models providing intermediate feedback on reasoning steps beyond terminal correctness.The survey contrasts process reward models with easier-to-implement outcome reward models, which can favor shallow heuristics and be susceptible to reward hacking.Source: §6.2 · p. 33

6.2.1 Algorithm Discovery and Reasoning

• RL fine-tuning combined with evolutionary search surpassed the FunSearch baseline on bin packing, traveling salesman, and flatpack algorithm tasks.The survey also reports improved scientific reasoning from RL fine-tuning based on expert genomics forum discussions converted into multiple-choice questions.Source: §6.2.1 · pp. 33, 35

6.2.2 Domain-Specific Reasoning and Optimization

• In domain-specific optimization, RL with protein language models generates sequences optimized for therapeutic or structural objectives, while molecular rewards target therapeutic properties and chemical validity.Reported examples include improved binding affinity and stability, including EGFR binder candidates, and molecular optimization balancing efficacy with validity.Source: §6.2.2 · p. 35

6.3 Hybrid and Preference Optimization Methods

• Hybrid approaches combine supervised grounding with RL refinement, while preference optimization can add diversity and novelty objectives to scientific generation.Standard DPO may favor safe, high-reward responses; CrPO and DivPO instead incorporate novelty, diversity, surprise, or rare high-quality responses into preference optimization.Source: §6.3 · pp. 35–36

6.4 Takeaways and Limitations

• CycleResearcher parameter-adapted models scored 5.15–5.38 overall versus 4.31 for the AI Scientist baseline, with acceptance rates of 24–35% versus 0%.The survey reports CrPO novelty scores of 6.8–7.9, compared with approximately 5.7 for SFT and 4.0 for base LLMs, and concludes that adaptation chiefly strengthens valueness while explicit exploration rewards yield stronger novelty.Source: §6.4 · p. 37
• SFT, RL, and hybrid methods respectively emphasize consistency, exploration, and balance, but face dataset bias, reward mis-specification, overfitting, and generalization challenges.These approaches internalize knowledge and creative pathways, potentially reducing reliance on test-time computation or external resources.Source: §6.4 · pp. 36–37

• Evaluation assesses final outputs and a model’s effectiveness as an ideator, whereas feedback guides generation through search and refinement.Source: §7 · p. 38
• Scientific ideas are difficult to evaluate because their impact unfolds over time and they are open-ended, while expert judgment, automated metrics, and LLM judges each have trade-offs.Source: §7 · p. 38
• The survey reviews computational and execution-based metrics, expert review, and LLM-as-a-judge models as three evaluation-framework families.Source: §7 · p. 39

7.1 Computational and Execution-Based Metrics

• Computational and execution-based metrics offer reproducible signals but only indirect, partial views of creativity, often missing conceptual depth and experts’ contextual insight.Source: §7.1 · p. 39

7.1.1 Novelty and Originality

• Novelty measures include absolute semantic distance, density-based measures, relative local density, and knowledge-graph triplets; together they capture complementary aspects of originality.Relative Neighbor Density achieved AUROC > 0.80 across computer science and biomedical datasets; triplet-based detection depends on reliable relation extraction and graph coverage.Source: §7.1.1 · pp. 39, 41

7.1.2 Diversity

• Novelty and diversity must be assessed jointly because models can produce unique ideas concentrated in a narrow conceptual area.APD measures set-level spread, CrPO rewards per-response dispersion, and Vendi Score is sensitive to redundancy.Source: §7.1.2 · p. 42

7.1.3 Feasibility and Resource-Efficiency

• Feasibility measures range from literature-based plausibility estimates to execution or simulation, with outcomes such as performance gain, reproducibility, and successful candidate screening.Execution-grounded evaluation is limited to fields with accessible simulators or experimental infrastructure, so scalable frameworks may combine it with LLM-as-a-judge evaluation.Source: §7.1.3 · p. 43

7.1.4 Utility and Impact

• Impact can be estimated with bibliometric or graph-based proxies, or assessed through experiments and behavioral outcomes; these measures do not capture all practical or theoretical significance.Execution-based impact assessment is constrained because many ideas are not directly executable and simulators exist only in select fields; human expertise remains important for nuanced creativity and long-term value.Source: §7.1.4 · pp. 43–44

• Experts can assess conceptual depth, originality, and methodological soundness that automated metrics struggle to approximate.Source: §7.2 · p. 44
• Human-review setups use ratings and qualitative feedback, including panels that assess creativity, feasibility, satisfaction, novelty, plausibility, and usefulness.One study assessed LLM-generated research ideas with over 100 domain experts, separating novelty and usefulness judgments.Source: §7.2 · pp. 44–45
• Consensual Assessment Technique aggregates multiple expert scores into consensus ratings, while AI Scientist v2 uses conference peer review as validation.Source: §7.2 · p. 45
• Human evaluation is time-consuming, costly, difficult to scale, and subject to variation among evaluators.Source: §7.2 · p. 45

7.3 LLM-as-a-Judge Evaluation

• LLM judges assess plausibility, novelty, and impact between automated metrics, execution-based validation, and human review, but do not replicate domain expertise or epistemic reasoning.Source: §7.3 · p. 45

7.3.1 Prompt-based / Inference-time judges

• Prompt-based and retrieval-augmented judges provide low-cost, reproducible evaluations; the Idea Novelty Checker achieved over 13% higher agreement with expert judgments.Their reliability remains sensitive to prompt design, retrieval coverage, and base-model reasoning quality.Source: §7.3.1 · p. 47

7.3.2 Supervised fine-tuned judges

• DeepReview-14B surpasses CycleReviewer-70B, GPT-o1, and Deepseek-R1 by dynamically scaling inference complexity.Source: §7.3.2 · p. 48
• SFT-based judges learn scoring patterns from review annotations, but can over-rely on spurious cues and lose robustness when those cues fail.Source: §7.3.2 · p. 48

7.3.3 Reinforcement-optimized judges

• HARPA-generated hypotheses outperform strong AI-researcher baselines in feasibility (+0.78) and groundedness (+0.85) on human ratings, with 20 versus 11 successful CodeScientist runs.Source: §7.3.3 · p. 48
• J1, ReviewRL, and REMOR exemplify RL-trained judges that improve evaluation behavior, including cross-domain generalization, reliability, coherence, factuality, and fairness.Source: §7.3.3 · p. 48

7.3.4 Hybrid Frameworks

• Post-trained reviewers may produce confident but uninformed judgments when evaluations require current or specialized scientific evidence.Source: §7.3.4 · p. 49
• Hybrid evaluators combine learned judgment with retrieval and inference-time reasoning; DeepReviewer-14B outperforms larger models with fewer tokens, while LLM judges still need human calibration for fairness and nuance.Source: §7.3.4 · p. 49

7.4 Challenges in Evaluation

• Scientific idea evaluation must resolve novelty, feasibility, diversity, and impact, making the Product dimension the central bottleneck.The measurable outcome is where these criteria must ultimately be judged.Source: §7.4 · p. 49
• Human ratings vary with evaluator expertise and expectations, while human and automated evaluators can reproduce biases against AI-generated or unconventional ideas.Embedding-similarity measures and LLM critics can also reflect training-data biases or shift with prompting.Source: §7.4 · pp. 49, 51
• Studies use different metrics, task framings, evaluation protocols, and operational definitions, making results difficult to reproduce or compare across benchmarks.Table 6 illustrates variation in task framing and evaluation protocols; the cited prose also describes differences in how novelty and feasibility are operationalized.Source: Table 6 · p. 50, §7.4 · p. 51
• Most benchmarks assess final artifacts rather than the generative process, relying on short-term or proxy measures while leaving longer-term scientific impact largely unexamined.Treating generation as a black box limits direct assessment of creativity’s source or mechanism.Source: §7.4 · p. 51
• Expert review offers nuance but is costly and fatigue-prone, whereas automated metrics scale efficiently but miss the depth of human judgment.The survey calls for transparent standardized metrics, scalable proxies, and carefully calibrated human oversight.Source: §7.4 · p. 51

8 Ethical Concerns

• Automated evaluators may mistake coherent, innovative-looking ideas based on incorrect assumptions or unsound reasoning for credible science.The survey describes this as plausible pseudo-science that automated evaluators may struggle to detect.Source: §8 · p. 52
• LLM-based ideation and evaluation may amplify literature biases, favoring dominant paradigms while neglecting underexplored or unconventional directions.The passage notes that these systems may reinforce prevailing scientific dogmas.Source: §8 · p. 52
• Prompt injection embedded in manuscripts can manipulate automated reviews, exposing evaluation pipelines to systematic exploitation.The passage cites demonstrated vulnerabilities in LLM-based reviewers.Source: §8 · p. 52
• Generated ideas or procedures in chemistry and biology raise dual-use concerns if safeguards are insufficient.The survey also identifies unresolved governance challenges around verification robustness, bias, and adversarial manipulation.Source: §8 · p. 52

9.1 Embracing the Open-Ended Nature of Scientific Discovery

• Scientific discovery has evolving goals and an unbounded hypothesis space, while many current methods assume fixed objectives and closed task spaces.Future systems should support continuous exploration, novelty seeking, and dynamic reframing to generate new questions as well as candidate answers.Source: §9.1 · p. 53

9.2 Establishing Standardized Benchmarks and Robust Evaluation Frameworks

• Shared benchmarks should cover multiple domains and abstraction levels, while human evaluation should transparently report evaluator backgrounds, guidelines, and agreement.Peer-review-style evaluation is proposed as one route to structured human feedback.Source: §9.2 · pp. 53–54
• Automated evaluation should be modular and criterion-specific, with retrieval for novelty, domain-specific executors or simulators for feasibility, and expert models for correctness or coherence.Online retrieval and test-time reasoning are proposed to ground judgments in current literature; trained evaluators may also learn superficial cues.Source: §9.2 · p. 54

9.3 Building Domain-Rich Simulators for Feasible and Scalable Training

• Domain-rich simulators could provide safe, rapid feedback for hypothesis testing where real scientific environments are costly, slow, or unsafe.The survey cites AtomAgents for materials design and Robin for biological assays and robotic wet-labs.Source: §9.3 · p. 54
• Domain-specific simulators could support structured rewards, scalable pretraining, and safe exploration, helping LLMs generate, test, and refine hypotheses at scale.The survey describes these environments as strengthening the Person and Press dimensions through feedback-rich settings.Source: §9.3 · p. 55

9.4 Beyond Next-Token Prediction: Architectural Bottlenecks for Creativity

• Next-token prediction favors local coherence and incremental extrapolation, which may constrain the bold conceptual leaps scientific discovery requires.The survey proposes exploring alternatives such as multi-token prediction, state-space models, diffusion-based language models, and hybrids.Source: §9.4 · p. 55

10 Conclusion

• LLMs can retrieve, combine, and refine scientific concepts to accelerate hypothesis formation, but transformational discovery remains elusive.The conclusion characterizes combinatorial and exploratory creativity as increasingly within reach.Source: §10 · p. 56
• The survey organizes methods into knowledge augmentation, prompt-driven creativity, inference-time search, multi-agent collaboration, and parameter-level adaptation.It also reviews a patchwork of automated metrics and ad hoc human studies, with reproducibility and comparability remaining serious challenges.Source: §10 · p. 56
• Current methods emphasize Process and Press, while Person and Product remain underdeveloped, including intrinsic generative capacities and transparent evaluation.The conclusion links these gaps to moving beyond proof-of-concept ideation toward scientific contributions.Source: §10 · p. 56