PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Ideation: Designing LLM Agents to Generate Novel Research Ideas on Scientific Concept Network

Keyu Zhao, Weiquan Lin, Qirui Zheng, Fengli Xu, Yong Li

arXiv:2511.02238v1cs.AI

TL;DR

Existing ideation approaches miss contextual relationships between scientific concepts or fail to dynamically use the scientific concept network. Deep Ideation combines that network with iterative LLM-driven refinement and reviewer-trained feedback; it outperformed the best baseline by an average of 10.25% across four AI domains.

  • Problem

    Prior methods rely on keyword co-occurrence or semantic similarity and overlook contextual concept relationships, while some LLM methods do not dynamically use the evolving scientific concept network.

    Source: §1 · p. 2

  • Method

    Deep Ideation connects LLM agents to a literature-derived scientific network for iterative keyword expansion and idea refinement, with an Idea Stack and a reviewer-feedback-trained critic.

    Source: §1 · p. 2

  • Results

    Deep Ideation outperformed the best baseline by an average of 10.25% across four AI domains and reached the acceptance level of eight out of ten AI conferences.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Human evaluation found that the generated proposals provided genuine inspiration and value to researchers.

    Source: §1 · p. 2

Abstract

from arXiv · show

Novel research ideas play a critical role in advancing scientific inquiries. Recent advancements in Large Language Models (LLMs) have demonstrated their potential to generate novel research ideas by leveraging large-scale scientific literature. However, previous work in research ideation has primarily relied on simplistic methods, such as keyword co-occurrence or semantic similarity. These approaches focus on identifying statistical associations in the literature but overlook the complex, contextual relationships between scientific concepts, which are essential to effectively leverage knowledge embedded in human literature. For instance, papers that simultaneously mention "keyword A" and "keyword B" often present research ideas that integrate both concepts. Additionally, some LLM-driven methods propose and refine research ideas using the model's internal knowledge, but they fail to effectively utilize the scientific concept network, limiting the grounding of ideas in established research. To address these challenges, we propose the Deep Ideation framework to address these challenges, integrating a scientific network that captures keyword co-occurrence and contextual relationships, enriching LLM-driven ideation. The framework introduces an explore-expand-evolve workflow to iteratively refine research ideas, using an Idea Stack to track progress. A critic engine, trained on real-world reviewer feedback, guides the process by providing continuous feedback on the novelty and feasibility of ideas. Our experiments show that our approach improves the quality of generated ideas by 10.67% compared to other methods, with ideas surpassing top conference acceptance levels. Human evaluation highlights their practical value in scientific research, and ablation studies confirm the effectiveness of each component in the workflow. Code repo is available at https://github.com/kyZhao-1/Deep-Ideation.

1 INTRODUCTION

Prior ideation methods miss contextual concept relationships or fail to use evolving scientific networks dynamically. Deep Ideation combines a literature-derived network, iterative idea refinement, and reviewer-trained feedback; experiments report gains across four AI domains.

1 INTRODUCTION

  • Semantic-embedding approaches overlook co-occurrence-based relationships, while LLM ideation methods do not dynamically retrieve and integrate evolving, context-dependent concept connections.Extracting meaningful relationships from literature and enabling dynamic LLM interaction with the network are both described as challenging.Source: §1 · p. 2
  • Deep Ideation iteratively explores network relationships, expands the keyword set, and refines proposals while an Idea Stack tracks progress.The framework integrates LLM agents with a scientific network to retrieve and incorporate concept relationships through evolutionary search and refinement.Source: §1 · p. 2
  • Deep Ideation outperformed the best baseline by an average of 10.25% across four AI domains and reached the acceptance level of eight out of ten AI conferences.Human evaluation found the proposals offered genuine inspiration and value to researchers.Source: §1 · p. 2
  • The framework builds a scientific concept network from approximately 100,000 papers from ten major AI conferences over the past decade, using keyword co-occurrence and relationships.The network dataset is intended to be made publicly available.Source: §1 · p. 2
  • A critic model trained on real-world reviewer feedback guides ideation with expert-level evaluation.The framework fine-tunes an LLM using publicly available review data to provide scientifically grounded feedback.Source: §1 · p. 2

2 PROBLEM FORMULATION

The paper formulates a scientific network as a graph of keywords linked by co-occurrence, with edge features aggregating their relationships across papers. It defines ideation as transforming initial keywords into an idea proposal, and represents that proposal as depending on keywords, system configuration, and external knowledge.

2.1 DEFINITION OF SCIENTIFIC NETWORK

  • A scientific network is a graph whose keyword nodes are connected when the keywords co-occur in at least one paper.An edge feature represents the relationship between its keywords across the papers in which both appear.Source: §2.1 · p. 3
  • Each edge feature F_ij aggregates or processes the pair's relationships across papers where both keywords appear.The paper denotes that set of papers as P_i,j and the paper-level relation as relation(v_i, v_j, p).Source: §2.1 · p. 3

2.2 FORMULATING RESEARCH IDEATION PROBLEM

  • Scientific ideation transforms an initial set of domain keywords into an idea proposal that innovatively synthesizes those concepts.The definition frames ideation as generating and refining ideas based on existing knowledge.Source: §2.2 · p. 3
  • The proposal is formulated as I = f(K, Θ, Ψ), depending on input keywords, system parameters or workflow, and accessible external knowledge.K denotes the keyword set, Θ the model parameters or workflow, and Ψ the external knowledge base.Source: §2.2 · p. 3
  • Figure 2 depicts the framework with a maximum keyword-set size of four.Source: Figure 2 · p. 4
  • The ideation process is iterative: the keyword set is refined using the previous proposal and external knowledge.At iteration t, I_t denotes the proposal, K_t the keyword set, and Ψ_t the external knowledge base.Source: §2.2 · p. 4
  • The iterative cycle continues until the idea reaches satisfactory novelty and feasibility or a stopping criterion is met.Source: §2.2 · p. 4

3 METHODS

Deep Ideation connects LLM agents to a scientific network for iterative idea generation, with an Idea Stack tracking progress and a Review Model guiding refinement. Its workflow combines network-based keyword exploration and selection, idea formulation, and evolution of keywords or proposals.

3.1 OVERVIEW

  • The framework iteratively queries a scientific network of keyword co-occurrence, tracks idea progress in an Idea Stack, and uses a Review Model to evaluate novelty and feasibility.The Review Model provides evaluative feedback to guide ideation.Source: §3.1 · p. 4

3.2 KEY COMPONENTS OF DEEP IDEATION FRAMEWORK

  • The four components construct a network from paper keywords and their contextual relationships, analyze relations, select impactful keywords, and formulate them into structured, scientifically grounded proposals.The proposal includes research background, a research idea, and a general implementation approach.Source: §3.2 · pp. 4–5

3.3.1 EXPLORE

  • Explore identifies up to m neighboring terms per initial keyword and aggregates relationship information from papers where keyword pairs co-occur.The network edge relationship is aggregated across the papers containing both keywords.Source: §3.3.1 · p. 5

3.3.2 EXPAND

  • Expand selects a keyword based on its relationships to the current set, adds it, and formulates an updated proposal using the Idea Stack’s record of prior iterations.The Idea Stack records keyword evolution, idea development, and evaluations.Source: §3.3.2 · pp. 5–6

3.3.3 EVOLVE

  • Evolve begins when the keyword set reaches Lmax; a Router directs whether keywords or the proposal should evolve, incorporating prior insights, new findings, and emerging trends.The keyword set is dynamically replaced, while proposals are refined to adapt to the evolving research context.Source: §3.3.3 · p. 6

3.4 CRITIC MODEL

  • The Critic Model guides iterative refinement with evaluative feedback, using training data structured from simulated reviewer reasoning about novelty and feasibility.The authors motivate this simulation by noting that direct LLM review can produce surface-level assessments lacking deep, domain-specific understanding (Zhao et al., 2025a).Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Across four AI domains, Deep Ideation outperforms the best baseline on average scores, and human evaluation also favors its proposals. Ablations describe the roles of iterative keyword evolution and critic feedback, while case studies illustrate generated proposal mechanisms.

4.1 EXPERIMENTAL SETUP

  • The setup uses about 100,000 papers across DL, NLP, CV, and General AI, compares several discovery methods and accepted papers, and averages evaluations from five advanced models.GPT-4o-mini runs the framework components, while Qwen3-8B is used for the critic.Source: §4.1 · p. 7

4.2.1 LLM AS JUDGE EVALUATION RESULTS.

  • 10.92% in DL, 9.48% in NLP, 13.91% in CV, and 9.48% in General AI: Deep Ideation's Avg. score gains over the best-performing baseline.It also surpasses the level of many accepted AI conference papers in several domains.Source: §4.2.1 · p. 7, Table 1 · p. 7

4.2.2 HUMAN EVALUATION RESULT

  • Human evaluation with 54 researchers found that Deep Ideation consistently outperformed the best baseline on idea value, innovation, and structure.One participant described the proposals as innovative and grounded in existing scientific knowledge.Source: §4.2.2 · p. 8, Table 2 · p. 8

4.3 ABLATION STUDY OF DEEP IDEATION

  • Removing Evolve lowers performance across domains as keywords cannot adapt to review feedback; without the critic, idea evolution lacks evaluative guidance.The authors associate missing critic feedback with less coordinated exploration and weaker novelty and feasibility.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 CASE STUDY

  • Case studies illustrate proposals combining face detection and segmentation with reinforcement learning, and using NCE in reinforcement learning to improve preference robustness.The right-hand proposal cites a relevant paper to refine its generated idea.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

5 RELATED WORKS

Related work spans LLM systems for scientific research and methods specifically targeting scientific ideation. These approaches use varied combinations of retrieval, iteration, validation, and knowledge resources.

5.1 LLMS FOR SCIENTIFIC RESEARCH

  • ResearchAgent combines LLMs with symbolic knowledge graphs for experimental design, AutoSurvey automates literature surveys, and AlphaEvolve evolves code-level hypotheses.Their printed references are Baek et al. (2024), Wang et al. (2024b), and Novikov et al. (2025), respectively.Source: §5.1 · p. 9

5.2 LLMS FOR SCIENTIFIC IDEATION

  • SciMON and ResearchAgent iteratively refine hypotheses using new literature, while maximizing semantic novelty to improve novelty and relevance.The cited works are Wang et al. (2024a) and Baek et al. (2024).Source: §5.2 · p. 9
  • Other ideation methods include multi-agent validation in SciAgents, patent-graph mining in MOOSE-Chem, zero-shot hypothesis generation, and CycleResearcher's self-supervised review loop.The passage cites Ghafarollahi & Buehler (2025), Yang et al. (2024), Qi et al. (2023), and Weng et al. (2024), respectively.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion presents Deep Ideation as a framework combining LLMs, scientific networks, iterative refinement, and reviewer-feedback-based evaluation. Experiments across AI domains show improvements in novelty and feasibility.

6 CONCLUSION

  • Deep Ideation integrates LLMs with scientific networks, an Idea Stack for iterative refinement, and a review model trained on real-world feedback to assess novelty and feasibility.Experiments across AI research domains demonstrate improvements in both novelty and feasibility.Source: §6 · p. 9

A.1 PROMPT FOR DEEP IDEATION

The prompts use the Idea Stack and scientific-network relationships to guide keyword selection, idea formulation, and iterative refinement, while evaluating proposals for novelty and feasibility using keyword and graph features.

A.1.2 PROMPT OF KEYWORD SELECTION

  • Keyword selection uses the Idea Stack’s history of ideas, scores, and keyword changes to choose one connected candidate that advances the current idea without repeating past directions.The selection prompt weighs novelty and feasibility alongside relevance: longer network paths may offer diversity, while shorter paths may help ground feasibility, but path length alone should not decide.Source: §A.1.2 · pp. 14–15
  • Candidates are linked to a current keyword and may include shortest-path distances that indicate their proximity to the existing keyword set.The output identifies the selected keyword, its connected current keyword, and why it supports the idea’s development.Source: §A.1.2 · p. 15

A.1.3 PROMPT OF KEYWORD REPLACEMENT

  • The replacement prompt considers previous replacements and their reasons, and limits candidate replacements to those that can replace a flexible keyword.The candidate’s shortest paths indicate its relation to the existing set; the prompt also cautions against redundant or conflicting research directions.Source: §A.1.3 · pp. 15–16
  • Keyword replacement chooses one flexible keyword to replace with a connected candidate, using the Idea Stack to improve the idea while avoiding redundant or conflicting past directions.Longer shortest paths may support novelty and shorter paths may support feasibility; the output specifies the replacement, its connection, and the rationale.Source: §A.1.3 · p. 16

A.1.4 PROMPT OF IDEA FORMULATION

  • Idea formulation combines the provided keywords into a coherent proposal that builds on the Idea Stack and includes research background, the idea, and a general implementation approach.The prompt asks the proposal to integrate new keywords coherently and avoid ideas redundant with past directions.Source: §A.1.4 · pp. 17–18

A.1.5 PROMPT OF REVIEW MODEL

  • The review prompt evaluates research proposals using their associated keywords and scientific-network graph features along two dimensions.The network represents keywords as nodes and co-occurrence in a scientific paper as an edge; listed features include neighbor count, connectivity, and shortest paths.Source: §A.1.5 · p. 18
  • The review prompt scores novelty and feasibility from 1–5 and asks the model to consider keyword-network features, including neighbor count, connectivity, and shortest paths.It requests a score and short description for each dimension.Source: §A.1.5 · pp. 18–19

A.1.6 PROMPT OF ROUTER

  • The router receives the current idea, keywords, and novelty and feasibility assessments, then chooses between keyword replacement and idea rewriting.Its required output includes the action and a brief reason.Source: §A.1.6 · pp. 19–20
  • The router directs rewriting when keywords are strong but the proposal is poorly written, and replacement when low scores indicate keywords need improvement.It asks the decision-maker to use novelty and feasibility assessments alongside keyword quality.Source: §A.1.6 · p. 19

A.1.7 PROMPT OF KEYWORD REPLACE

  • Replacement candidates connect to a current keyword and the wider keyword set; shorter paths to other keywords indicate greater relevance, and candidates can replace only flexible keywords.The prompt supplies candidate relationships and shortest-path lengths for the current set.Source: §A.1.7 · pp. 20–21
  • The replacement prompt uses an Idea Stack to record each round’s keywords, idea, scores, keyword changes and reasons, and refined idea.The history is used to trace how the research direction evolves and to avoid redundant or overly similar directions.Source: §A.1.7 · p. 20
  • The prompt instructs the model to use the Idea Stack to preserve research progression, avoid repeating past directions, and choose replacements that improve the idea.It asks the model to consider prior replacements and their reasons.Source: §A.1.7 · p. 20
  • For novelty, the prompt favors a potentially longer-path keyword; for weak feasibility, it favors a shorter-path keyword, while requiring focus alignment and avoiding redundant or conflicting directions.Path length is not sufficient on its own to justify a novelty-oriented replacement.Source: §A.1.7 · p. 21
  • The model must select one flexible keyword and report its replacement, the connected keyword, the replaced keyword, and the reason tied to progress in the Idea Stack.The rationale should explain how the replacement strengthens the idea and builds on prior work.Source: §A.1.7 · p. 21

A.1.1 PROMPT OF RELATION ANALYSIS

The prompt asks an assistant to analyze a paper containing two supplied keywords and explain how the paper constructs their connection. The explanation is limited to 2–4 sentences.

A.1.1 PROMPT OF RELATION ANALYSIS

  • The prompt frames the task as analyzing academic papers.Source: §A.1.1 · p. 14
  • The assistant receives two keywords and a paper in which both appear, then explains how the paper constructs their connection.The provided paper input includes its introduction.Source: §A.1.1 · p. 14
  • The requested explanation must be limited to 2–4 sentences.Source: §A.1.1 · p. 14

A.2 DATASET DETAILS

The dataset contains 107,443 papers from major AI conferences over the past decade, grouped into four categories. Papers supply keywords for a co-occurrence network, and generated proposals are classified by the domain of their initial keyword.

A.2 DATASET DETAILS

  • The dataset contains 107,443 papers from major AI conferences spanning the past decade.The conferences are ICLR, NeurIPS, ICML, ACL, NAACL, CVPR, ICCV, AAAI, and IJCAI.Source: §A.2 · p. 21
  • The corpus is grouped into DL (ICLR, NeurIPS, ICML), NLP (ACL, NAACL), CV (CVPR, ICCV), and General AI (AAAI, IJCAI).Source: §A.2 · p. 21
  • Three to four keywords are extracted from each paper to construct a scientific network based on keyword co-occurrence.Source: §A.2 · p. 21
  • An initial keyword is selected from a domain, and the resulting idea proposal is classified by that domain.Source: §A.2 · p. 21

A.3 BASELINE METHODS

The benchmark set includes methods for scientific discovery and idea generation, alongside accepted papers from major AI conferences. The compared approaches span literature-grounded refinement, knowledge graphs, domain-specific hypothesis generation, and review-agent workflows.

A.3 BASELINE METHODS

  • Benchmarks include Sci. Net. Emb., SciMON, SciAgents, and MOOSE-Chem (Yang et al., 2024).Their stated approaches draw on human expertise, literature-based refinement, ontological knowledge graphs and multi-agent systems, and structured chemistry hypothesis generation, respectively.Source: §A.3 · p. 22
  • The comparison also includes zero-shot hypothesis proposers (Qi et al., 2023) and ResearchAgent (Baek et al., 2024).The former examines proposals without prior fine-tuning; the latter combines iterative generation with LLM-based review agents.Source: §A.3 · p. 22
  • Accepted papers from major AI conferences serve as baselines, represented by title, abstract, and introduction organized into an idea-proposal format.Source: §A.3 · p. 22

A.4 EVALUATION DETAILS AND MODEL FINE-TUNING

The analysis identifies maximum neighborhood size 12 and keyword set size 4 as the strongest reported settings, while describing trade-offs at smaller or larger values. It also reports the model and human-evaluation procedures.

A.4 EVALUATION DETAILS AND MODEL FINE-TUNING

  • GPT-4o-mini powers four ideation modules, while a LoRA-fine-tuned Qwen3-8B reviews ideas for novelty and feasibility.The reviewing model was fine-tuned on 4,278 examples.Source: §A.4 · p. 22
  • Final performance scores average evaluations from GPT-4o, Gemini-2.5-Flash, Grok-3, DeepSeek-V3.1, and Qwen3-235B-A22B.Source: §A.4 · p. 22

A.5 ANALYSIS

  • Maximum neighborhood size controls sampled knowledge breadth per keyword, while maximum keyword set size determines how many keywords generate a proposal.Figure A.5 reports performance as the sum of novelty and feasibility.Source: §A.5 · p. 22, Figure A.5 · p. 23

A.5.1 EFFECT OF MAX NEIGHBORHOOD SIZE.

  • Deep Ideation performs best with a maximum neighborhood size of 12.Smaller neighborhoods limit surrounding knowledge; sizes above 12 may overload prioritization and dilute idea focus. Figure A.5 measures performance as novelty plus feasibility.Source: §A.5.1 · p. 22, Figure A.5 · p. 23

A.5.2 EFFECT OF MAX KEYWORD SET SIZE

  • A maximum keyword set size of 4 improves performance by balancing idea diversity and depth with focus and logical connection.Smaller sets limit breadth and diversity; overly large sets can produce disjointed ideas and forced connections.Source: §A.5.2 · p. 23, Figure A.5 · p. 23

A.6 HUMAN EVALUATION DETAILS

  • Human participants rated proposals on novelty and feasibility and described selected ideas that particularly caught their interest.Source: §A.6 · p. 23
Loading 2511.02238v1…