PaperlayerView exact revision on arXiv

Source-linked AI summary

Spark: A System for Scientifically Creative Idea Generation

Aishik Sanyal, Samuel Schapiro, Sumuk Shashidhar, Royce Moon, Lav R. Varshney, Dilek Hakkani-Tur

arXiv:2504.20090v2cs.AIcs.IRcs.LG

TL;DR

Scientific idea generation has historically relied on human creative intuition, and evaluating the creativity of generated ideas calls for rigorous critique. SPARK combines literature-informed idea generation with JUDGE, a review-trained evaluator; the system has produced and filtered over 10,000 AI-focused research ideas, and the authors release its training dataset for further exploration.

Abstract

from arXiv · show

Recently, large language models (LLMs) have shown promising abilities to generate novel research ideas in science, a direction which coincides with many foundational principles in computational creativity (CC). In light of these developments, we present an idea generation system named Spark that couples retrieval-augmented idea generation using LLMs with a reviewer model named Judge trained on 600K scientific reviews from OpenReview. Our work is both a system demonstration and intended to inspire other CC researchers to explore grounding the generation and evaluation of scientific ideas within foundational CC principles. To this end, we release the annotated dataset used to train Judge, inviting other researchers to explore the use of LLMs for idea generation and creative evaluations.

Introduction

Scientific idea generation has historically depended on human creative intuition, while recent work shows that LLMs can generate research ideas. SPARK pairs idea generation with a review-trained evaluator and releases its training dataset to support research grounded in computational creativity.

Introduction

  • Scientific idea generation requires creative intuition but has historically been restricted to human scientists.Source: Introduction · p. 1
  • Recent research has demonstrated that LLMs can generate novel research ideas, and idea generation remains a crucial first step in autonomous research systems.The passage also notes systems producing full-length papers at near-human quality or workshop-length papers that passed human peer review.Source: Introduction · p. 1
  • SPARK combines scientific idea generation with JUDGE, an evaluator trained on 600K expert human reviews from OpenReview.The system is intended to evaluate the creativity of SPARK’s proposed ideas.Source: Introduction · p. 1
  • The authors release the OpenReview dataset used to train JUDGE and frame SPARK around computational creativity principles.The paper focuses on idea generation as the first stage of scientific discovery.Source: Introduction · p. 1

Related Work

Prior scientific-ideation systems use retrieval, iterative refinement, or multiple agents, while some broader research automation systems also generate and validate hypotheses or produce papers. SPARK is presented as a demonstration that grounds scientific idea generation in computational creativity principles and integrates retrieval, generation, and simulated review.

Related Work

  • Galactica used scientific corpora for technical tasks including hypothesis generation, but encountered factual-grounding and hallucinated-reference issues despite strong benchmark performance.Source: Related Work · p. 1
  • SciPIP combines semantic retrieval and citation awareness but is primarily evaluated in NLP, while SciMON iteratively maximizes novelty but often yields low practical feasibility and experimental utility.Source: Related Work · p. 1
  • VirSci uses multiple LLM agents and improves idea diversity and research-trend alignment, but, like SciMON, focuses on novelty without directly optimizing experimental usefulness.Source: Related Work · p. 1
  • AI Scientist generates and experimentally validates hypotheses but requires human oversight, while Zochi produces workshop-accepted papers across domains with substantial human intervention and computational resources.Source: Related Work · p. 1
  • SPARK demonstrates grounding scientific idea generation in computational creativity principles by integrating literature retrieval, idea generation, and simulated peer-review evaluation.Source: Related Work · pp. 1–2

Methods

SPARK brings together specialized AI-driven modules for literature retrieval and idea synthesis, refinement, and evaluation. The system is designed to support computational creativity in scientific idea generation.

Methods

  • SPARK integrates AI-driven components to support computational creativity in scientific idea generation.Source: Methods · p. 2
  • Specialized agent-based modules handle literature retrieval and the synthesis, refinement, and evaluation of ideas.Source: Methods · p. 2
  • Figure 1 illustrates how the SPARK and XPLOR systems interact.Source: Methods · p. 2

Literature Retrieval via XPLOR

XPLOR gathers literature through embedding-based retrieval, relevance scoring, and diversity-aware re-ranking. Its recursive agent loop refines searches using intermediate summaries and can provide evidence to SPARK’s idea-generation stage.

Literature Retrieval via XPLOR

  • XPLOR uses OpenAI embeddings, FAISS, MMR, and a recursive agent-based retrieval loop to gather and refine research documents for idea generation.Source: Literature Retrieval via XPLOR · p. 2

Text Embedding and Retrieval:

Relevance Scoring and Maximal Marginal Relevance:

Recursive Retrieval and Evidence Gathering Loop:

  • XPLOR iteratively refines database queries using LLM-generated intermediate summaries until depth and coverage criteria are met.One example criterion is accumulating at least five evidence snippets from multiple sources; interactive mode takes user queries, while autonomous mode takes queries from the SPARK Idea Generator.Source: Recursive Retrieval and Evidence… · p. 2
  • The pipeline connects XPLOR retrieval to seed-proposal synthesis and then to JUDGE critique and ranking.Source: Figure 1 · p. 2

Idea Generation with SPARK Idea Generator

SPARK’s Idea Generator uses XPLOR search results to identify open problems and synthesize structured, contextually relevant research proposals. If further refinement is needed, XPLOR can retrieve additional literature for a specific area of improvement.

Idea Generation with SPARK Idea Generator

  • The Idea Generator identifies open problems related to the query and synthesizes structured, contextually relevant, well-motivated ideas from XPLOR search results.Source: Idea Generation with SPARK Idea… · p. 2
  • XPLOR recursively retrieves relevant papers and identifies open research problems from a user’s research question.The resulting concepts and problems shape a problem-driven research prompt.Source: Idea Generation with SPARK Idea… · p. 2
  • An LLM uses the prompt to produce a proposal with a reasoning plan, novel concepts, title, and abstract.The output is intended to be contextualized within existing literature; XPLOR can retrieve more literature to address a specific area for improvement.Source: Idea Generation with SPARK Idea… · p. 2

Idea Evaluation with SPARK Filter

SPARK Filter simulates peer review to assess research ideas’ viability and quality. Its workflow generates multiple critiques, synthesizes a decision, and has filtered more than 10,000 AI-focused ideas.

Idea Evaluation with SPARK Filter

  • The Filter is SPARK’s evaluative module, simulating peer review to assess proposed ideas’ viability and quality.Each idea’s title and abstract are processed by the JUDGE model, which generates multiple critiques highlighting strengths, limitations, and areas for improvement.Source: Idea Evaluation with SPARK Filter · pp. 2–3
  • A separate LLM synthesizes multiple reviews into an ACCEPT or REJECT classification, an explanation, and a utility score.Source: Idea Evaluation with SPARK Filter · p. 3
  • Over 10,000 AI-focused research ideas have been generated and filtered, demonstrating scalability for large-scale computational creativity tasks.Source: Idea Evaluation with SPARK Filter · p. 3

Evaluator Training: The JUDGE Model

JUDGE is trained to critique scientific ideas using academic standards, with data transformations intended to isolate conceptual assessment from empirical outcomes. The model learns review and abstract mappings and, on designated tasks, both model sizes outperform their respective base models and other contemporary state-of-the-art LLMs.

Evaluator Training: The JUDGE Model

  • JUDGE is a purpose-built model for critiquing scientific ideas according to academic standards, addressing the need for rigorous novelty and creativity evaluation.The paper says off-the-shelf LLMs trained with SFT or RLHF tend to prioritize agreeableness over critical scrutiny.Source: Evaluator Training: The JUDGE Model · p. 3

Data Acquisition and Preprocessing

  • Idea abstracts omit specific results, implementation details, and performance metrics while retaining the problem, approach, and claimed novelty; idea reviews focus on those conceptual aspects.The dataset pipeline harvested abstract–review pairs; the authors chose abstracts over full papers to isolate and sharpen assessment of underlying ideas, and used DeepSeek-V3 to transform original abstract–review pairs for reproducibility.Source: Data Acquisition and Preprocessing · p. 3

Multi-Task Training Framework

  • JUDGE is trained on four mappings: original and idea abstracts to their reviews, and original and idea reviews back to their abstracts.Training combines forward evaluation and backward generation mappings; it uses continued pretraining and LoRA to update base-model weights.Source: Multi-Task Training Framework · pp. 3–4

Evaluation and Model Scaling

  • Both JUDGE versions consistently outperform their respective base models and other contemporary state-of-the-art LLMs on the designated evaluation tasks.Evaluation uses a held-out temporal OpenReview partition and assesses generated-review quality and explicit review-score prediction; the Qwen-72B-based version provides roughly 10× the parameters of the Qwen-7B-based version for scaling analysis.Source: Evaluation and Model Scaling · p. 4

Discussion and Limitations

SPARK combines literature retrieval, multiagent idea generation, and a simulated peer-review filter in a modular proof of concept. The authors identify design refinements and an AI-only evaluation scope, and release the JUDGE training dataset to support community experimentation.

Discussion and Limitations

  • SPARK decouples retrieval, multiagent idea generation, and simulated peer review into a test bed for exploring how generation and evaluation strategies interact.Source: Discussion and Limitations · p. 4
  • SPARK has been evaluated only in a controlled setting for idea generation in AI; evaluation in additional scientific fields is needed to assess its abilities across disciplines.The authors also identify one-pass filtering, similarity-based retrieval, and critiques without explicit explanatory traces as design choices for further exploration.Source: Discussion and Limitations · p. 4
  • The authors released the annotated papers-and-reviews dataset used to train JUDGE to facilitate community experimentation.They propose comparative studies of retrieval methods, generation prompts, reviewer-agent designs, and creativity-evaluation strategies using SPARK as a foundation.Source: Discussion and Limitations · p. 4

Looking Ahead: Computational Creativity

The section raises open questions about evaluating scientific ideas and using AI to generate transformational ideas. It argues these questions merit further exploration by computational creativity researchers.

and Scientific Idea Generation

Creative Criteria for Ideas Versus Papers

  • Idea utility must be assessed before research outcomes are observed, and theory-guided plus data-driven approaches may be fruitful.The released dataset may help facilitate data-driven utility evaluation.Source: Creative Criteria for Ideas Versus… · p. 5

Transformative Idea Generation

  • SPARK uses concept extraction and problem synthesis to construct relevant ideas, often by combining existing concepts; whether AI can discover transformational ideas remains open.The authors argue that this direction merits considerable future exploration given transformational creativity's historical impact on science and society.Source: Transformative Idea Generation · p. 5

Conclusion

The paper presents SPARK as a system demonstration combining scientific idea generation with peer-review-inspired evaluation. It offers the framework, data resources, and observations as a starting point for future computational creativity research.

Conclusion

  • SPARK unifies literature retrieval, LLM-driven idea generation, and a peer-review-inspired filter.Source: Conclusion · p. 5
  • The system demonstration is presented as a concrete step toward grounding scientific idea generation in computational creativity principles.The authors aim to inspire work on LLM-based idea generation, evaluation, and feasibility.Source: Conclusion · p. 5

Related papers

Loading 2504.20090v2…