PaperlayerView exact revision on arXiv

Source-linked AI summary

IDEAgent: Agentic Quality-Diversity Search for Research Idea Generation

Varun Gumma, Navonil Majumder, Soumitra Sinhahajari, Soujanya Poria

arXiv:2607.22375v1cs.AI

TL;DR

Scientific ideation must produce ideas that are both high-quality and sufficiently diverse, while evaluating quality or diversity alone does not capture that joint objective. IDEAgent manages ideas through lineages and quality improvement, and across 32 topics in 8 Computer Science domains it achieved higher average Yield and non-zero Yield on more topics than the considered baselines.

  • Problem

    Scientific ideation needs to produce multiple high-quality, sufficiently diverse ideas, while separate quality and diversity measures can fail to capture that joint objective.

    Source: §1 · p. 2

  • Method

    IDEAgent frames ideation as Quality-Diversity search, managing idea lineages with quality evaluations and repair or refinement, alongside archives of completed, rejected, and historical ideas.

    Source: §1 · p. 2, §8 · p. 12

  • Results

    Across 32 topics in 8 Computer Science domains, IDEAgent achieved higher average Yield and more topics with non-zero Yield than all considered baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis found improvements in soundness and clarity while preserving non-obviousness across evaluations by internal and external judges.

    Source: §1 · p. 2

    The study evaluates idea portfolios rather than real-world feasibility, exact correctness, or practical effectiveness, and does not establish publishability or absence from existing literature.

    Source: §9 · p. 13

Abstract

from arXiv · show

Large Language Models (LLMs) have significantly automated the process of scientific discovery over the past few years. However, existing systems share one core limitation: they generate and optimize ideas independently for either Quality or Diversity. This often leads to the generation of ideas in close proximity to one another or to a large set of trivial, unsound, or unclear concepts. In this work, we instead argue that research ideation should be treated as a conjunction of both objectives and framed as a Quality-Diversity (QD) search. In line with this perspective, we introduce IDEAgent, a multi-agent framework that manages the evolution of ideas through lineages. We jointly drive Quality using multi-objective feedback for dedicated repair and refinement, while Diversity is achieved through lightweight sequential memory and explicit comparison against completed ideas, their historical ancestors, and rejected proposals. To systematically evaluate this QD conjunction, we develop Yield, a joint metric that computes the largest set of mutually diverse ideas that satisfy a predetermined quality threshold. Finally, through evaluations across 32 topics spanning 8 domains of Computer Science, we show that IDEAgent outperforms the best baseline by 3.89x on Yield, while achieving non-zero Yield on 8x more topics. We further corroborate these findings through an analysis of quality improvements, showing that repair and refinement are crucial for building logical rigor and clarity while preserving non-obviousness. To encourage future research on QD-search-based ideation, we open-source IDEAgent at https://github.com/declare-lab/IDEAgent.

1 Introduction

IDEAgent frames scientific ideation as a Quality-Diversity search for ideas that are individually sound, clear, and non-obvious while remaining mutually distinct. It combines lineage-based idea development with Yield, a metric for counting viable, diverse ideas.

1 Introduction

  • Existing ideation systems optimize quality or diversity independently, leading to repeated concepts, paraphrased variations, and diminishing returns across successive generations.The paper argues that useful ideation should produce multiple independent research directions, not just a plausible individual proposal.Source: §1 · pp. 1–2
  • IDEAgent develops ideas through lineages and uses compact memories and dedicated archives to compare proposals with completed ideas, historical variants, and rejected directions.Specialized agents evaluate quality; accepted ideas may be refined, while near-misses receive targeted repairs for flaws and inconsistencies.Source: §1 · pp. 2–3
  • Yield applies hard thresholds for non-obviousness, soundness, and clarity, then counts the largest subset of ideas that satisfies pairwise diversity constraints.It is designed to measure how many ideas from a generated set are viable for a human researcher to pursue.Source: §1 · pp. 2–3
  • Across 32 topics in 8 Computer Science domains, IDEAgent achieves higher average Yield and non-zero Yield on more topics than the considered baselines.Internal and external evaluators also find improvements in soundness/rigor and clarity while non-obviousness is preserved.Source: §1 · pp. 2–3
  • The paper formulates automated scientific ideation as a joint search for multiple high-quality, sufficiently diverse research directions.It presents IDEAgent as an open-source multi-agent framework for this task.Source: §1 · p. 2

2 Related Work

The section traces autonomous scientific-discovery systems, identifies novelty as a bottleneck in AI ideation, and situates the paper against QDAIF in text-generation quality-diversity search.

Autonomous Frameworks for Scientific Discovery

Novelty Bottlenecks in AI-based Ideation

  • AI ideation often clusters around seed literature and recombines familiar variations, while open-ended exploration remains difficult despite divergence methods; early ideation diversity bounds downstream engineering success.The conceptual-narrowness finding is reported by Tang and Yang (2026), divergence methods by Hu et al. (2025) and Radensky et al. (2026), and the downstream-task limitation by Audran-Reiss et al. (2025).Source: §2 · p. 3

Quality-Diversity Search in Text Generation

  • Compared with QDAIF’s mutation, elitism, and fixed archive grid, this work uses grid-free evolution and directed feedback to improve idea quality.QDAIF is the closely related approach identified by Bradley et al. (2024).Source: §2 · p. 3

3 Methodology

IDEAgent treats research ideation as a budgeted Quality–Diversity search, evolving structured idea lineages through quality assessment, diversity checks, repair, and refinement. Its procedure preserves distinct mechanisms while improving promising proposals.

3.1 Task Formulation as Quality–Diversity Search

  • Each seed defines one discovery attempt: ideas are structured proposals, and close variants retain the same lineage rather than counting as separate discoveries.The proposal specifies a problem, causal diagnosis, intervention, mechanism, assumptions, expected effect, and evaluation strategy; diversity is judged from the problem and causal mechanism rather than fixed search-space axes.Source: §3.1 · p. 3

3.2 Agents and Search Memory

  • The agent team structures drafts for comparison, evaluates quality, soundness, and diversity, and turns detected failures into focused feedback for repair or refinement.The Stenographer summarizes each idea as problem, mechanism, novel value addition, assumptions, and expected measurable effect; evaluators assess non-obviousness, clarity, feasibility, logical consistency, and archive-relative diversity.Source: §3.2 · p. 4
  • Compact active, repair, historical, and rejected-pattern memories guide new seed prompts, while full parent text is supplied when its lineage is revised.Qualified ideas that leave the active archive are retained historically; failures are aggregated into patterns for explicit avoidance rather than stored as exact discarded proposals.Source: §3.2 · p. 5

3.3 Search Procedure

  • Duplicate matches are not new discoveries: historical-only matches and ambiguous matches are rejected, while a candidate matching one active idea may replace it if its combined quality is higher.A candidate with no duplicate but insufficient diversity is discarded; a sufficiently diverse, unmatched candidate may enter the active set if it passes qualification. Accepted ideas compete for limited archive capacity, with feasibility breaking quality-score ties.Source: §3.3 · pp. 5–6
  • Candidates must pass quality and diversity requirements; near-misses may receive one repair, while accepted ideas can be refined under additional safeguards.The primary search sets non-obviousness, soundness, and clarity thresholds and a diversity floor at 60; feasibility is measured but excluded from the qualification gate. Refinement replaces a parent only if the child passes qualification and diversity checks, improves combined quality, and wins a blinded non-obviousness comparison.Source: §3.3 · pp. 5–6
  • The primary configuration runs 10 fresh seeds with active capacity 10 and allows up to two auxiliary drafts per seed for repair and refinement.At most one repair and two accepted refinements are allowed per seed; the total number of Ideator calls is 10–30.Source: §3.3 · p. 6

4 Background Corpus Collection

IDEAgent builds a shared background corpus by tracing influential citations from published papers, grouping related citation sets, and selecting diverse, identifiable topics.

4 Background Corpus Collection

  • The corpus begins with published papers from multiple arXiv domains, whose influential citations provide a shared seed knowledge bank.The stated goal is to generate diverse, non-trivial, sound research ideas from a shared set of background papers.Source: §4 · p. 6
  • Papers are grouped when their influential-citation sets have an Intersection-over-Union above 0.5, and each group’s union becomes its background-paper set.The procedure first filters papers by publication status and top-k% ranking.Source: §4 · p. 6
  • Only sets supporting at least three published papers are retained; topics must contain 5–8 background papers and receive an LLM-generated descriptive tag.Each topic represents a uniquely identifiable sub-domain or task.Source: §4 · p. 6
  • With a budget of Γ = 6, greedy maximum-diversity selection produces the final topic collection presented in Table 3.Source: §4 · pp. 6–7
  • Hyperparameters were not rigorously optimized because of budget constraints, so reported results could be under-reported or unoptimized.Source: §4 · p. 6

5 Evaluation

IDEAgent’s evaluation treats ideation as a set-level Quality-Diversity task: ideas must be individually high-quality and mutually diverse. Its primary metric, Yield, counts the largest subset that passes quality thresholds and pairwise diversity constraints.

Evaluation protocol

  • Evaluation scores each topic’s returned idea set for both per-idea quality and pairwise diversity, rather than assessing ideas in isolation.An LLM judge supplies independent quality scores and pairwise diversity metrics.Source: §5 · p. 7

Quality Axis

  • Quality is judged on five axes: non-obviousness and soundness are primary, while mechanism clarity, feasibility, and significance are secondary.Non-obviousness concerns originality; soundness concerns whether the mechanism works and its reasoning is rigorous; clarity concerns whether it is specified well enough to implement. Feasibility and significance are judged separately from correctness and practicality, respectively.Source: §5 · p. 7

Diversity Axis

  • Diversity is scored pairwise using semantic signatures across eight axes, with disagreements on failure mode, causal diagnosis, and intervention weighted more heavily.A judge assigns same, variant, related, or distinct relations per axis, then uses those relations to produce a pairwise score from 0 to 9.Source: §5 · p. 7

Yield

  • Yield counts the maximum number of ideas meeting non-obviousness, soundness, and clarity thresholds whose every pair meets the diversity threshold.It finds the largest qualifying subset as a maximum clique, so near-identical high-quality ideas count as one rather than many.Source: §5 · p. 8

Successful Topics

  • A topic is successful for a method if it yields at least Φ ideas that pass the Yield gates; the reported measure is the proportion of successful topics.The paper says a general idea generator should ideally have this proportion at 1.Source: §5 · p. 8

6 Experimental Setup

Section 6 describes the agent models and evaluation setup, then defines four baselines that vary in how ideas are generated, remembered, and selected.

6.1 Agents

  • Initial experiments found that smaller open-source models often lacked logical consistency and misjudged ideas, producing low soundness scores and unreliable evaluations.Source: §6.1 · p. 8
  • Experiments use gpt-5.6-sol6 as Ideator, gemini-3.1-pro7 for Critic, Soundness Panel, and Quality Judge, and flash models for Diversity Judge and Steno.Claude Sonnet 5 and Opus 4.7 provide external judgments; their scores are averaged into one jury verdict. All baselines receive an idea budget of B=10.Source: §6.1 · p. 8

6.2 Baselines

  • Stateless generates B ideas in parallel without correlation or knowledge of the other ideas.Source: §6.2 · p. 8
  • One-Shot generates all B ideas in one output, allowing a thinking-enabled Ideator to use shared thinking space and condition ideas on preceding ideas.The setup is described as a rudimentary form of stateful generation.Source: §6.2 · p. 8
  • Sequential-Memory generates ideas independently while carrying forward prior ideas’ core signatures, but omits IDEAgent’s repair and refinement routines.Source: §6.2 · p. 8
  • NOVA-inspired SM runs three rounds of B generations, uses a judge to retain three seed concepts per round, then selects B distinct ideas from the full pool.It excludes NOVA’s online retrieval and, like SM, performs no repair or refinement.Source: §6.2 · pp. 8–9

7 Results and Analysis

Across 32 topics, IDEAgent achieves higher Yield and broader coverage than the baselines, while repair and refinement improve idea quality. The two judges generally agree on rankings, with agreement varying by rubric.

IDEAgent has a Higher Yield

  • 3.6 versus 1.9: IDEAgent’s Yield exceeds Sequential-Memory’s at (C≥6, Dij≥7), while the advantage remains +0.9 at (NB≥7, S≥7).At (NB≥6, S≥6), the reported gain is +1.9; both methods’ Yield approaches zero at NB≥8 and S≥8.Source: §7 · pp. 9–10, Figure 2 · p. 10

IDEAgent vs. Baselines

  • Against NOVA, IDEAgent’s Yield improvement is 2× at NB≥6 and 3.89× at NB≥7; Non-Obviousness, Soundness, and Clarity each achieve their highest scores under IDEAgent.Source: §7 · p. 10
  • NOVA achieves the highest pairwise diversity, but One-Shot has zero Yield; the authors suggest shared memory can propagate weak ideas, while compact sequential signatures are more computationally efficient over long runs.The authors hypothesize lightweight sequential core memory induces meaningful diversity while limiting cross-contamination between ideas.Source: §7 · p. 10
  • 31 of 32 versus 25 of 32 topics: IDEAgent versus the best baseline yields at least one qualifying idea at NB≥6; at NB≥7, the counts are 27 of 32 versus 8 of 32.For at least two qualifying ideas at NB≥6, NOVA succeeds on 11 of 32 topics and IDEAgent on 23 of 32.Source: §7 · p. 10

Quality improvements due to repair and refinement

  • 28 of 30 repair candidates qualified after improvement, and refinement replaced the original idea in 82% of cases.Among 320 lineages, 30 required repair and 182 underwent refinement after qualifying.Source: §7 · pp. 10–11
  • +23.2: repair’s maximum Soundness improvement; refinement’s largest average rubric gain was +8.9 in Clarity, while both processes improved Non-Obviousness by about +7.Internally assessed repair increased Clarity by +16.7; repair’s maximum Yield gain was about 0.97 at both non-obviousness thresholds, while refinement’s Yield gains were comparatively small.Source: §7 · p. 11
  • +1.63 and +1.20: external judges’ Clarity gains from repair and refinement, respectively; refinement increased Yield by +0.88 and +0.78 at NB thresholds 6 and 7.External judges agreed with internal judges on the direction of improvement, though their absolute gains differed; the section does not directly correlate score deltas across the different scales.Source: §7 · p. 11, Figure 3 · p. 11

Inter Annotator Agreement

  • 0.60: judges’ highest absolute-score agreement was for Clarity, compared with 0.26 for Soundness; rank correlations ranged from 0.409 for Soundness to 0.761 for Feasibility.Feasibility had low absolute-score agreement (κ=0.210) but strong rank consistency (ρ=0.761); agreement was generally higher for more concrete rubrics than for holistic judgments.Source: §7 · pp. 11–12, Table 2 · p. 12

8 Conclusion

IDEAgent frames scientific ideation as a Quality-Diversity search, combining quality improvement and lineage-inspired memory to build portfolios of ideas with high quality density.

8 Conclusion

  • IDEAgent sequentially generates and evaluates ideas, using a repair or optional refinement subroutine to address identified opportunities.Its lineage-inspired process manages completed, rejected, and historical ideas in dedicated archives enabled by core signature comparison.Source: §8 · p. 12
  • Yield jointly evaluates quality and diversity by filtering ideas at fixed quality thresholds and selecting the largest subset that meets a pairwise diversity threshold.Source: §8 · p. 12
  • IDEAgent was evaluated across 32 Computer Science research topics, with analysis of the contributions of its quality improvement loop and stateful memory.The authors also open-sourced the experimental framework to support a new perspective on AI-driven scientific ideation.Source: §8 · p. 12

9 Limitations · A List of Domains & Topics · B Evaluation: Rubrics and Algorithms

IDEAgent’s evaluation and scope have important limitations: its assessments rely on LLM judges, its ideas are not validated through implementation or literature searches, and computational and model choices constrain the experiments. The authors argue that the framework and Yield metric can accommodate alternative models and evaluation protocols.

9 Limitations

  • All evaluations rely primarily on LLM judges, with two independent models equally weighted and five soundness scores aggregated; Yield can also pair with human judgments.The authors note that alternative evaluation protocols can be used with the model-agnostic metric.Source: §9 · p. 12
  • The study evaluates portfolios of diverse, high-quality ideas, not their real-world feasibility, exact correctness, practical effectiveness, publishability, or novelty against existing literature.Validating these claims would require implementation and end-to-end experiments, expert validation, and comprehensive retrieval over published and emerging research; the authors report soundness and logical consistency across rigorous evaluations.Source: §9 · p. 13
  • Each idea receives only one repair and at most two refinements, since these computationally heavy rounds involve all agents and additional proprietary-LLM calls would raise costs.The authors aim to demonstrate follow-up repair and refinement as methods for improving idea quality.Source: §9 · p. 13
  • Experiments use proprietary LLM APIs because smaller open-source models lacked knowledge, idea formulation, refinement, and evaluation capacity; the framework is model-agnostic, and the authors expect its principles to transfer.The expected transfer concerns sequential generation, quality improvements through targeted multidimensional evaluation, and feedback.Source: §9 · p. 13
  • Preliminary tests of Gemma-4-31B, Qwen-3.6-27B, GPT-OSS-120B, Gemini 3.1 Pro Preview, Claude Sonnet 5, and Claude Opus 4.8 found diverse, non-obvious ideas but frequent serious soundness flaws.Those flaws substantially reduced overall idea quality, according to evaluators.Source: §9 · p. 13
  • Claude Fable 5 was excluded from the main experiments because of budget constraints despite producing moderately sound, highly non-obvious, and diverse ideas.In a cs.AI-001 pilot, its mean soundness score was 66.4 versus 91.8 for GPT-5.6-sol; it had a nine-point non-obviousness advantage but scored three points lower in diversity, requiring more repairs and contributing to its exclusion.Source: §9 · p. 13

C Additional Results · D Cost and Futher Scaling

Table 3 lists the domains and topics included in the dataset.

D Cost and Futher Scaling

  • Table 3 lists the domains and topics included in the dataset.Source: Table 3 · p. 16

D.1 Cost Metrics

The cost metrics define generation cost by method and topic for a required yield, with CST reported as NA when no topic meets that yield. The evaluation uses Φ = 2 as its principal diverse-portfolio criterion because pairwise diversity requires at least two qualifying ideas.

D.1 Cost Metrics

  • CST is reported as NA when no topic reaches the required yield; Table 1 evaluates 32 topics with (l, m, τ) = (7, 6, 7), k ∈ {6, 7}, and Φ ∈ {1, 2, 3}.C_M,T denotes method M's generation cost for topic T.Source: §D.1 · p. 16
  • Φ = 2 is the principal diverse-portfolio criterion because pairwise diversity is non-vacuous only when at least two qualifying ideas are produced.Source: §D.1 · p. 16

D.2 Analysis

IDEAgent is cost-effective when success requires multiple diverse, high-quality ideas, and scaling fresh generations alone does not reproduce its Yield advantage.

Scaling the Baselines

  • On a four-topic subset, Sequential Memory’s Yield at NB≥6 tripled from 10 to 30 generations but did not improve at 45, while its NB≥7 performance rose then declined.This exploratory result, alongside per-topic distributions, suggests that additional fresh sampling alone does not reproduce IDEAgent’s performance or explain its gains solely through a larger generation budget.Source: §D.2 · p. 17
  • Scaling Stateless from 10 to 50 generations raised mean Yield by approximately 27% at NB≥6 and 62% at NB≥7, yet IDEAgent achieved 2.06× and 2.46× its Yield, respectively.The baselines were scaled to approximately match IDEAgent’s generation budget, with up to 10 ideas retained from each generation pool.Source: §D.2 · p. 17, Figure 5 · p. 21

CST

  • At two qualifying diverse, high-quality ideas per topic, IDEAgent is approximately 1.6× cheaper than the strongest baseline and succeeds on 2.1× as many topics at NB≥6 and 8× as many at NB≥7.At a depth of three qualifying ideas, IDEAgent is approximately 11× cheaper than the only baseline that succeeds; Stateless is most cost-effective when just one idea qualifies, a criterion that does not test pairwise diversity.Source: §D.2 · p. 17, Table 14 · p. 23

Related papers

Loading 2607.22375v1…