Source-linked AI summary
IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation
Varun Gumma, Navonil Majumder, Soumitra Sinhahajari, Soujanya Poria
TL;DR
Existing automated ideation systems typically optimize either quality or diversity, leaving the joint objective insufficiently addressed. IDEAgent frames ideation as Quality-Diversity search and, across 32 topics in 8 Computer Science domains, produces higher average Yield and more topics with non-zero Yield than baselines.
Problem
Existing automated ideation systems separately optimize quality or diversity, motivating evaluation of portfolios containing multiple high-quality, mutually distinct research ideas.
Method
IDEAgent uses multi-agent idea lineages with quality repair and refinement, diversity archives, and Yield to jointly evaluate quality and pairwise distinctness.
Results
IDEAgent achieves higher average Yield and more topics with non-zero Yield than all considered baselines across 32 topics spanning 8 Computer Science domains.
Takeaways & Limitations
The results support Quality-Diversity search as a way to build portfolios of diverse, high-quality research ideas while improving soundness, clarity, and preserved non-obviousness.
Takeaways & Limitations
The study does not evaluate real-world feasibility, exact correctness, practical effectiveness, publishability, or absence from existing literature.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have significantly automated the process of scientific discovery over the past few years. However, existing systems share one core limitation: they generate and optimize ideas independently for either Quality or Diversity. This often leads to the generation of ideas in close proximity to one another or to a large set of trivial, unsound, or unclear concepts. In this work, we instead argue that research ideation should be treated as a conjunction of both objectives and framed as a Quality-Diversity (QD) search. In line with this perspective, we introduce IDEAgent, a multi-agent framework that manages the evolution of ideas through lineages. We jointly drive Quality using multi-objective feedback for dedicated repair and refinement, while Diversity is achieved through lightweight sequential memory and explicit comparison against completed ideas, their historical ancestors, and rejected proposals. To systematically evaluate this QD conjunction, we develop Yield, a joint metric that computes the largest set of mutually diverse ideas that satisfy a predetermined quality threshold. Finally, through evaluations across 32 topics spanning 8 domains of Computer Science, we show that IDEAgent outperforms the best baseline by 3.89x on Yield, while achieving non-zero Yield on 8x more topics. We further corroborate these findings through an analysis of quality improvements, showing that repair and refinement are crucial for building logical rigor and clarity while preserving non-obviousness. To encourage future research on QD-search-based ideation, we open-source IDEAgent at https://github.com/declare-lab/IDEAgent.
1 Introduction
The paper frames automated scientific ideation as a Quality-Diversity search requiring multiple high-quality, mutually distinct research directions rather than independent optimization of quality or diversity. It introduces IDEAgent and Yield to operationalize and evaluate this joint objective across 32 topics spanning 8 Computer Science domains.
- Method: IDEAgent is a multi-agent framework that navigates QD search by developing ideas sequentially against compact memories and preserving idea lineages.Its process combines conceptual diversification with rigorous multi-agent quality evaluation and comparisons against completed, historical, and rejected ideas.
- Evaluation: Yield jointly measures the Quality and Diversity of an idea set, addressing the failure of either objective alone to distinguish useful exploration from near-clones or trivial proposals.The metric is designed to capture mutually distinct, non-obvious, sound, and clear ideas at the group level.
- Evaluation: 32 research topics across 8 Computer Science domains are used to evaluate whether IDEAgent consistently generates denser sets of high-quality ideas within a limited ideation budget.The evaluation compares average Yield and the number of topics with non-zero Yield against considered baselines.
- Motivation: Automated ideation should produce multiple high-quality ideas that are sufficiently diverse to maximize pursuable research directions.The paper argues that quality and diversity must be optimized jointly rather than independently.
- Method: IDEAgent uses dedicated multi-dimensional evaluations, lifecycle-specific archives, and targeted repair and refinement subroutines to improve idea quality and novelty assessment.Compact core summaries support efficient novelty comparison without requiring full textual ideas.
2 Related Work
Prior work has advanced LLM-based scientific discovery toward autonomous multi-agent pipelines, but AI ideation remains constrained by conceptual narrowness. QDAIF is the closest related quality-diversity approach, while this work distinguishes its evolutionary setup from fixed-grid search.
- Autonomous Frameworks for Scientific Discovery: LLM scientific-discovery systems have progressed from drafting assistance to autonomous, end-to-end research pipelines.Early frameworks also introduced structured multi-agent environments to improve hypothesis quality.
- Autonomous Frameworks for Scientific Discovery: Collaborative debate among distinct agent personas produces more robust proposals than single-turn generation.
- Novelty Bottlenecks in AI-based Ideation: LLM ideation agents often cluster around seed literature and familiar technical variations instead of pursuing genuinely new research paths.This failure reflects a broader bottleneck in divergent thinking.
- Quality-Diversity Search in Text Generation: QDAIF is the closest related LLM quality-diversity method, using evolutionary mutations, elitist candidate evaluation, and a fixed archive grid.
- Quality-Diversity Search in Text Generation: This work instead uses a more amorphous evolutionary setup without a fixed grid search space or MAP-Elites.
3 Methodology
IDEAgent frames scientific ideation as a Quality–Diversity search that evolves structured research proposals through bounded lineages. Its multi-agent pipeline evaluates, repairs, routes, and archives ideas to retain high-quality, mutually distinct discoveries.
- QD formulation: IDEAgent seeks as many high-quality and mutually distinct ideas as possible under a fixed discovery budget, with each idea specifying its problem, mechanism, assumptions, effects, and evaluation strategy.Ideas are represented as structured research proposals rather than unconstrained text.
- Lineage evolution: Ideas evolve within lineages: repairing or refining a seed preserves its lineage, and close variants cannot count as separate discoveries or consume additional discovery credit.A seed and its lineage represent one fresh attempt to discover a mechanism and consume exactly one unit of budget.
- Agent pipeline: Each draft is summarized as a quintuple containing its problem, central mechanism, novel value addition, key assumptions, and expected measurable effect for interpretable pairwise comparison.The Stenographer produces this compact representation from loosely formed proposals.
- Agent pipeline: Three evaluators score non-obviousness, mechanism clarity, feasibility, logical consistency, and diversity, while a Critic converts scores and failure evidence into focused repair or refinement feedback.The Quality Evaluator uses a 0–100 scale for non-obviousness, clarity, and feasibility; the Diversity Judge also identifies closest mechanisms.
- Routing and archives: The controller routes candidates using active, historical, repair, and failed/rejected memories, rejecting duplicates or insufficiently diverse ideas while allowing qualified near-misses a repair opportunity.Historical summaries preserve valid ideas that leave the active archive, including replaced parents and displaced ideas.
- Routing and archives: A new mechanism enters the active archive when capacity is available, or replaces the weakest active idea only when its combined quality score Q_b is higher, with feasibility breaking ties.Accepted ideas that leave the active archive are retained in the historical archive as valid, good-quality pieces.
4 Background Corpus Collection
The background corpus is constructed by grouping influential citations across arXiv papers into shared knowledge banks, filtering for demonstrated usefulness, and selecting diverse topics. Each retained topic contains 5 to 8 background papers and receives an LLM-generated descriptive tag.
- Corpus construction: Papers from cs.AI, cs.CL, and cs.RO are used to build a background corpus serving as a shared seed knowledge bank for diverse, sound research ideas.The collection begins with published papers from multiple arXiv domains.
- Corpus construction: Papers are grouped when the Intersection-over-Union (IoU) of their influential citations exceeds 0.5, with each group’s citation union defining its background knowledge bank.Papers are first filtered by publication timeframe, status, and ranking before citation-based grouping.
- Topic filtering: Only sets supporting at least three published papers are retained, ensuring each knowledge bank has demonstrated support for multiple novel, high-quality research ideas.This filtering step is intended to remove knowledge banks without evidence of repeated usefulness.
- Topic selection: Topics containing between 5 and 8 background papers are retained, assigned descriptive tags by an LLM, and processed with greedy maximum diversity selection.The final topic collection is presented in Table 3.
- Limitations: The study does not rigorously optimize hyperparameters because of budget constraints, so reported results may be under-reported or unoptimized.This caveat is stated in a footnote accompanying the corpus-collection section.
5 Evaluation
IDEAgent is evaluated as a Quality-Diversity search over sets of ideas, requiring individually high-quality ideas that are also mutually diverse. The evaluation operationalizes quality through multiple axes, diversity through pairwise semantic comparison, and performance through Yield and successful-topic rates.
- Evaluation framing: Each topic is evaluated on its returned set of N ideas, requiring every idea to be non-obvious, sound, and clear while the set remains mutually diverse.This set-level evaluation avoids rewarding redundant high-quality ideas or diverse but trivial, invalid, or unclear ideas.
- Quality Axis: Quality is assessed using non-obviousness, soundness, mechanism clarity, feasibility, and significance.Non-obviousness and soundness are emphasized together because an original idea with inconsistent reasoning is unusable.
- Diversity Axis: Diversity decomposes each idea into eight semantic axes and compares every pair using same/variant/related/distinct relations across those axes.The core axes—failure mode, causal diagnosis, and intervention—receive greater weight than supporting axes.
- Yield: Yield(NB ≥ k, S ≥ l, C ≥ m, D ≥ τ) counts the maximum number of sound, clear, non-trivial, and mutually diverse ideas surviving predefined quality thresholds and a maximum-clique diversity constraint.The procedure first filters ideas by non-obviousness, soundness, and clarity, then extracts the largest subset whose pairs meet the diversity threshold.
- Successful Topics: A topic is successful at yielding Φ when the tested method generates at least Φ ideas satisfying the Yield gates.The evaluation reports the proportion of successful topics, with an ideal general generator reaching a ratio of 1.
6 Experimental Setup
Experiments use proprietary large-scale models after smaller open-source models showed poor logical consistency and unreliable internal evaluation. The setup compares stateless, one-shot, sequential-memory, and NOVA-inspired sequential-memory generation strategies, with IDEAgent’s repair and refinement distinguished from memory-only baselines.
- Model Selection: Smaller open-source models produced severely low-soundness ideas and frequently misjudged generations during internal evaluation.These failures motivated using proprietary large-scale models for all experiments.
- Baselines: Stateless generation produces B ideas in parallel without correlation or knowledge of one another.This is the simplest generation strategy used for comparison.
- Baselines: One-Shot generates all B ideas in a single output, allowing a thinking-enabled Ideator to share memory and condition later ideas on earlier ones.It is therefore analogous to a rudimentary stateful-generation strategy.
- Baselines: Sequential-Memory independently generates each of B ideas while carrying forward prior signature σ(i), but excludes repair and refinement.This makes it sequentially similar to IDEAgent while isolating lightweight memory’s contribution to diversification.
- Baselines: NOVA-inspired Sequential-Memory runs n=3 rounds, generates B ideas per round, and selects the best B from n·B candidates while emphasizing distinctness.It adds iterative seed-pool germination and replacement, omits online retrieval, and performs no subsequent repair or refinement.
7 Results and Analysis
IDEAgent achieves higher quality–diversity Yield and broader topic coverage than the baselines, while repair and refinement improve idea quality. External evaluations agree on improvement direction, and judge agreement is strongest for clarity and weakest for soundness.
- Yield: 3.6 Yield for IDEAgent versus 1.9 for Sequential-Memory at C≥6 and D_ij≥7, an improvement of approximately 50%.Yield declines toward zero for both methods under increasingly stringent thresholds, approaching zero when NB≥8 and S≥8.
- Baseline comparison: 3.89× Yield improvement over NOVA at NB≥7, following a 2× improvement at NB≥6.IDEAgent also achieves the highest Non-Obviousness, Soundness, and Clarity scores.
- Topic coverage: 31 of 32 topics yield at least one qualifying idea for IDEAgent versus 25 of 32 for the best baseline at D≥7, S≥7, C≥6, and NB≥6.At NB≥7, the gap widens, with the best baseline succeeding on 8 of 32 topics.
- Quality improvements: Soundness benefits most from repair and refinement, reaching nearly 100% qualification in both cases.Refinement starts from higher absolute scores because it applies only to ideas already meeting thresholds, whereas repair targets ideas narrowly missing them.
- Inter Annotator Agreement: 0.60 agreement for clarity versus 0.26 for soundness; rank correlations remain positive across all rubrics, ranging from 0.409 for soundness to 0.761 for feasibility.The results indicate that clarity is more readily identified consistently, while soundness judgments are more subjective.
8 Conclusion
IDEAgent reframes scientific ideation as a Quality-Diversity search for portfolios with maximum quality density. It combines sequential generation, multi-agent evaluation, quality improvement through repair or optional refinement, and lineage-inspired idea management.
- IDEAgent reframes scientific ideation as a Quality-Diversity search to build portfolios with maximum quality density.
- Sequential generation and multi-agent evaluation center the framework on the core requirements of Quality-Diversity search.
- A quality improvement subroutine repairs or optionally refines ideas based on identified opportunities.
9 Limitations · A List of Domains & Topics · B Evaluation: Rubrics and Algorithms
The paper identifies limitations involving LLM-based evaluation, restricted assessment scope, constrained repair and refinement budgets, and reliance on proprietary models. It does not evaluate real-world effectiveness, exact correctness, practical feasibility, or publishability of generated ideas.
- 9 Limitations: LLM-based judges assess non-obviousness, soundness, clarity, feasibility, and diversity, despite mitigation through independent models and aggregated soundness scores.Two independent judge models receive equal weight, while five independent soundness scores are aggregated.
- 9 Limitations: The evaluation targets portfolios of diverse, high-quality research ideas rather than real-world feasibility, exact correctness, or practical effectiveness.Those properties would require implementation, debugging, end-to-end experiments, and dedicated expert validation.
- 9 Limitations: The authors explicitly frame the work as limited to generating and evaluating diverse, high-quality research-idea portfolios.They do not claim that generated ideas are necessarily publishable or absent from existing literature.
- 9 Limitations: Each idea receives only one repair opportunity and at most two refinements because these computationally heavy operations involve all system agents.Using proprietary LLMs makes additional rounds substantially more expensive.
- 9 Limitations: All experiments use proprietary LLM APIs because current smaller open-source models lacked sufficient knowledge, idea formulation, refinement, and evaluation capacity.The framework is model-agnostic, so its sequential generation, targeted multidimensional evaluation, and feedback principles are expected to transfer across future models.
C Additional Results · D Cost and Futher Scaling
The reported additional-results material identifies the domains and topics included in the evaluation dataset.
- D Cost and Futher Scaling: The evaluation dataset comprises a specified list of domains and topics.Table 3 presents this list.
D.1 Cost Metrics
The section evaluates generation efficiency with Cost per Successful Topic (CST), dividing total cost across attempted topics by successful topics. It uses Φ = 2 as the principal criterion for diverse portfolios because pairwise diversity requires at least two qualifying ideas.
- Cost per Successful Topic: CST measures total generation cost across all attempted topics divided by the number of successful topics.It addresses whether cheaper methods can generate more ideas efficiently than fewer ideas from more advanced, expensive approaches.
- Evaluation settings: 32 topics are evaluated with (l, m, τ) = (7, 6, 7), k ∈ {6, 7}, and Φ ∈ {1, 2, 3}.CST is reported as NA when no topic reaches the specified yield.
- Diversity criterion: Φ = 2 is the principal diverse-portfolio criterion because pairwise diversity is non-vacuous only when at least two qualifying ideas are produced.The primary quality–diversity objective is to produce diverse, high-quality ideas for each topic.
D.2 Analysis
IDEAgent is more cost-effective when success requires multiple diverse, high-quality ideas, and scaled fresh-generation baselines still fail to match its Yield. Increasing generation budgets improves some baselines but can plateau or behave non-monotonically.
- CST: 1.6× cheaper per successful topic and 2.1× as many topics at NB≥6 make IDEAgent most cost-effective when success requires at least two diverse, high-quality ideas.Stateless remains most cost-effective when success requires only one qualifying idea, a criterion that does not exercise pairwise diversity.
- Scaling the Baselines: Stateless scales from 10 to 50 fresh generations across 32 topics, while Sequential Memory scales from 10 to 30 and 45 generations on four matched topics.Up to 10 ideas were retained from each generation pool using the same selection criteria.
- Scaling the Baselines: 27% and 62% mean-Yield improvements from scaling Stateless to 50 generations still leave IDEAgent at 2.06× and 2.46× its Yield at NB ≥6 and NB ≥7.Additional fresh generations improve Stateless but do not match IDEAgent’s ability to produce multiple qualifying ideas for the same topic.
- Scaling the Baselines: Threefold improvement at NB ≥6 occurs when Sequential Memory scales from 10 to 30 generations, with no further gain at 45 generations.At NB ≥7, performance increases at 30 generations before declining at 45, so the exploratory scaling trend is non-monotonic.