PaperlayerView exact revision on arXiv

Source-linked AI summary

IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

Jiarui Liu, Renjie Tao, Yiwei Liao, Chuanyang Jin, Kai Sun, Xiao Yang, Xinyuan Zhang, Xilun Chen, Zhuangqun Huang, Lechen Zhang, Yongjin Yang, Yinghui He, Weihao Xuan, Rakesh Wanga, Anuj Kumar, Mona T. Diab, Wen-tau Yih, Xin Luna Dong

arXiv:2610.04074v1cs.CL

TL;DR

Research ideation remains difficult to assess separately from downstream execution. IdeaScientist trains specialized agents to find research gaps, adapt mechanisms across fields, and write proposals using a decomposed idea corpus and temporally controlled evaluation. It outperforms the strongest open-source autoresearch baseline and the compared coding-agent systems on Qwen3.6-27B.

  • Problem

    Research ideation asks how to produce grounded proposals for open problems and how to evaluate them reliably before experimentation when many plausible solutions exist.

    Source: §1 · p. 1

  • Method

    IdeaScientist uses a gap finder, innovator, and report writer trained with role-specific reinforcement learning and supported by a decomposed literature vault.

    Source: §1 · pp. 2–3

  • Results

    14.0% higher overall score than the strongest state-of-the-art autoresearch baseline on Qwen3.6-27B; it also exceeds Claude Code SDK with Claude-4.8-Opus by 5.9% and Codex SDK with GPT-5.4 by 5.1%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Structured agent environments make scientific ideation amenable to controlled learning and evaluation rather than treating it as an opaque byproduct of end-to-end autoresearch.

    Source: §6 · p. 10

    Pre-execution evaluation must use proxies for eventual scientific value; direct execution is costly and mixes ideation with implementation skill.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Despite rapid progress in automating scientific research, generating promising and well grounded research solutions remains a central challenge. We isolate research ideation as a standalone task and build our solution on the intuition that a challenge in one field can often be addressed by a mechanism that solved an analogous challenge in another. Accordingly, we introduce IdeaScientist, which decomposes ideation into gap finding, innovation, and report writing, and trains each role with reinforcement learning. These roles identify limitations in related work, draw solution intuitions from analogous problem settings, and develop those intuitions into complete research proposals. To facilitate discovery of insights across domains, we construct the Svalbard Idea Vault, a corpus of 2.77M decomposed research ideas for retrieval, training, and temporally controlled evaluation. Our evaluation restricts access to literature available before a cutoff date and assesses how closely proposed directions align with those later explored in 15K papers authored by human researchers. On Qwen3.6-27B, IdeaScientist outperforms the strongest open-source autoresearch baseline by 14.0%, driven mainly by gains in novelty. On this 27B open backbone, IdeaScientist even outperforms Claude Code SDK with Claude-4.8-Opus and Codex SDK with GPT-5.4, by up to 5.9%.

1 Introduction

IdeaScientist treats research ideation as a distinct task: produce a grounded proposal for a challenge that current solutions fail to meet. It uses specialized agents and a decomposed idea corpus to generate and evaluate proposals, reporting gains over autoresearch and coding-agent baselines.

1 Introduction

  • Research ideation is defined as producing a grounded proposal from a problem definition and a challenge current solutions fail to meet.The challenge may concern an output desideratum or the process producing it, including quality, latency, or privacy risks.Source: §1 · p. 1
  • IdeaScientist separates gap finding, cross-field innovation, and report writing, training each role to identify limitations, adapt analogous mechanisms, and develop proposals.An orchestrator coordinates the agents, which use shared intermediate artifacts and the Idea Vault.Source: §1 · p. 2, Figure 1 · p. 2
  • The evaluation tests whether agents can anticipate research directions later pursued in human-authored papers, rather than search literature published after the cutoff.The held-out post-cutoff targets are drawn from 14,977 methodology papers.Source: §1 · p. 3
  • The Svalbard Idea Vault contains 2.77 million decomposed idea tuples and supports literature search, training-data generation, and evaluation.Evaluation uses post-cutoff targets while restricting literature access to papers available before January 1, 2026.Source: §1 · p. 3
  • 14.0% higher overall score than the strongest state-of-the-art autoresearch baseline; IdeaScientist also exceeds Claude Code SDK with Claude-4.8-Opus by 5.9% and Codex SDK with GPT-5.4 by 5.1%.Overall score averages novelty and proposal quality.Source: §1 · p. 3

2 Related Work

Prior work has explored autonomous research, generated-idea evaluation, and training agents for long-horizon tasks, but leaves open how to inspect and train literature search as a sequence of ideation operations.

2.1 Autonomous Research Systems and Research Ideation

  • Existing autonomous-research systems often entangle proposal quality with coding and experimental outcomes, while newer approaches do not train literature search as an inspectable sequence of research operations.IdeaScientist separately trains agents to find limitations in related work, seek mechanisms from other fields, and develop proposals.Source: §2.1 · p. 3

2.2 Evaluating Generated Research Ideas

  • Pre-execution evaluation uses proxies for scientific value, while LLM judges may disagree or favor their own model family and novelty can trade off against execution feasibility.Direct execution is more outcome-based but costly and mixes ideation with implementation skill; the authors validate their metrics through cross-evaluation and manual annotation.Source: §2.2 · p. 4

2.3 Training LLM Agents for Non-Verifiable, Long-Horizon Tasks

  • Proposal-level rubrics provide weak credit assignment and can invite reward hacking; IdeaScientist instead trains each role with a stage-matched human reference artifact and rubric before recomposing the roles.The method uses GRPO and then places the trained roles in the full agentic loop.Source: §2.3 · p. 4

3 IdeaScientist: A Framework for Autoresearch Ideation

IdeaScientist coordinates specialized agents that create intermediate artifacts for a grounded proposal, then trains the roles separately before recomposing them in a shared harness.

3.1 Harness Design

  • A gap finder identifies methodological limitations in closely related work, an innovator adapts mechanisms from other problem settings, and a report writer consolidates the intuition into a complete proposal.The proposal schema includes a problem setting, novelty, mechanism, related work, implementation detail, evaluation plan, falsifiable predictions, and limitations.Source: §3.1 · p. 4
  • The orchestrator maintains the evolving research plan and can revisit earlier stages, while separate paper readers and reviewers support targeted reading and proposal-quality checks.Intermediate artifacts allow the gap finder, innovator, and report writer to be checked and trained separately.Source: §3.1 · pp. 4–5

3.2 Training Design

  • Each role is trained separately with its own LoRA adapter using asynchronous GRPO, then recomposed with the same inputs, outputs, information access, and actions used at deployment.The orchestrator remains fixed because its narrow planning and scheduling action space has little room for improvement.Source: §3.2 · p. 5
  • Training uses a reference-grounded reward that combines artifact quality, agreement with a human reference, and citation overlap, subject to a validity gate and a protocol-violation penalty.The three components assess standalone quality, substantive research decisions, and overlap in cited papers.Source: §3.2 · p. 5

4 Dataset Construction

The Svalbard Idea Vault is an offline literature database whose decomposed papers provide role-specific artifacts and training examples from post-cutoff methodology papers.

4.1 Idea Vault construction

  • The Idea Vault collects arXiv papers across subject areas, covering 1986 through June 18, 2026, and contains 4.7 million papers, 2.8 million of them with full text.For papers collected on or after January 1, 2026, the authors also follow references one hop to add missing cited papers.Source: §4.1 · p. 5
  • Idea decomposition reconstructs each paper’s problem definition and challenge, then produces a methodological gap, research intuition, and complete proposal corresponding to the three agent roles.Source: §4.1 · p. 5

4.2 Training data construction

  • Training targets are methodological papers published on or after January 1, 2026, filtered for downloadable LaTeX and sufficient usable full text among cited works.The corpus excludes surveys, position papers, datasets and benchmarks, and purely empirical engineering reports.Source: §4.2 · p. 6
  • The filtered set contains 14,977 targets split into 14,500 training, 200 validation, and 277 held-out test papers, represented without measured experimental outcomes.The masked representations preserve problem, method, model design, related work, and limitations so the direction is available without its experimental success.Source: §4.2 · p. 6
  • Training examples use extracted gaps, intuitions, and proposals as a research-direction skeleton, while supporting claims and citations must be grounded in papers read through the environment.Artifacts are validated before inclusion in the training set.Source: §4.2 · p. 6

5 Experiments

The experiments compare proposal-generation systems on held-out research papers, using literature-based novelty and proposal-quality metrics. IdeaScientist-Trained achieves the strongest open-system overall score on Qwen3.6-27B, with its advantage driven mainly by novelty.

5.1 Experiment Setup

  • The evaluation uses a held-out 277-paper test split, compares general-purpose and autoresearch systems, and scores novelty and six proposal-quality dimensions.Systems receive the same problem definition, output schema, and offline literature database; target papers remain hidden and internet access is disabled.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Q1. Can Our Agents Generate High-Quality Research Proposals?

  • 74.6% overall, 14.0% above the best baseline on Qwen3.6-27B, makes IdeaScientist-Trained the strongest open system.The overall score averages novelty and proposal quality.Source: Table 1 · p. 7, §5.2 · p. 7
  • IdeaScientist-Base leads Claude Code SDK by 9.0% and Codex SDK by 2.4% in overall score, and ranks first among autoresearch systems on all three open backbones.Its margin over the strongest matched-backbone baseline ranges from 3.3% on GPT-OSS-120B to 12.9% on Qwen3.6-27B.Source: §5.2 · pp. 6–7
  • Training raises isolated held-out reward for the gap finder, innovator, and report writer by 10.1%, 10.7%, and 10.5%, respectively.Recomposing the trained roles improves average novelty by 1.9% and average proposal quality by 0.4% over Base.Source: §5.2 · p. 7
  • Up to +23.1% average novelty on Qwen3.6-27B drives IdeaScientist’s advantage; same-domain novelty gains reach +48.0%.The trained profile exceeds Base in five of six domain groups, and same-domain novelty gains are largest among the three literature scopes.Source: §5.2 · pp. 7–8, Figure 2 · p. 8
  • In blind human comparisons on 50 held-out papers, IdeaScientist-Trained beats every state-of-the-art system except DeepScientist by a statistically significant margin.It wins more often than IdeaScientist-Base, but that difference is not statistically significant.Source: §5.2 · p. 8, Figure 2 · p. 8

Case Studies.

Cross-domain retrieval improves novelty, while the case studies describe tighter links between failures and proposed mechanisms. Human comparisons and judge-agreement analyses support Qwen3.6-27B as a reasonable proposal-quality judge.

Case Studies.

  • Case studies find that IdeaScientist-Trained more tightly connects stated failures to mechanisms, making assumptions and expected effects easier to test than baseline control rules.The baselines’ plausible control rules have weaker justification for addressing the failure.Source: Case Studies · p. 8

5.3 Q2: How Much Does Each Element Contribute?

  • Adding the Gap Finder and Innovator raises All and Cutoff novelty, with the weaker Gemma-4-31B backbone benefiting more from the pipeline.The figure reports gains of up to 7.7% on Qwen3.6-27B and up to 13.0% on Gemma-4-31B.Source: §5.3 · p. 8, Figure 3 · p. 9
  • Cross-domain retrieval raises novelty by +5.3% (All), +3.7% (Cutoff), and +30.7% (In-domain, from 36.3% to 67.0%).It also raises Mechanism non-obviousness from 33.5% to 35.7% (+2.2%).Source: §5.3 · pp. 8–9, Figure 3 · p. 9

5.4 Q3: Can We Evaluate Research Proposal Quality Reliably?

  • Qwen3.6-27B judge scores correlate with human averages at r = 0.51–0.59 for the top three judges, comparable to inter-annotator agreement at r = 0.56.Qwen3.6-27B ranks second in consensus agreement, not significantly different from Claude-4.8, and has negligible self-preference bias lower than Claude-4.8’s.Source: §5.4 · p. 9, Figure 4 · p. 9

6 Conclusion

IdeaScientist’s conclusion frames research ideation as independently trainable and evaluable, while its appendices specify role permissions, retrieval practices, and fixed output schemas.

6 Conclusion

  • IdeaScientist isolates research ideation for training and evaluation independently of downstream implementation, and concludes structured agent environments can make ideation amenable to controlled learning.The system identifies limitations in related work, searches other settings for mechanisms, and develops them into grounded proposals.Source: §6 · p. 10

AI use statement

  • The authors report using generative AI tools only to polish phrasing and correct grammatical errors, with all edits reviewed by the authors.They state that they take responsibility for the final content.Source: AI use statement · p. 10

A.1 Action Space and Retrieval

  • The orchestrator plans and coordinates role agents, while producers and reviewers can retrieve papers, request readings, and write artifacts; only the paper reader sees full text.First-tier agents may request at most 20 readings, and receive a digest rather than the full text in their context.Source: §A.1 · pp. 15–16, Table 2 · p. 16
  • Retrieval instructions differ by role: gap finders search exact target-problem terminology, while innovators seek transferable mechanisms across multiple sub-fields.Reviewers search for same-problem prior art; report writers use whichever retrieval regime grounds the claims they must support.Source: §A.1 · p. 16

A.2 Agent Output Schemas

  • Producing roles use fixed schemas shared across training and deployment, and those schemas provide the surface for rewards and deterministic checks.The gap finder records prior work and up to two gaps per axis; the innovator appends candidate blocks, and the report writer produces a results-masked JSON proposal.Source: §A.2 · p. 16

B.1 Role-Specific Training and Rewards

The appendix motivates role-specific training as a remedy for sparse pipeline rewards and describes stage-specific inputs, validity checks, and reward components. It also reports choices made after comparing training alternatives.

B.1 Role-Specific Training and Rewards

  • The base model rarely completed the full pipeline, leaving most examples without informative reward; separate role training avoids this cold start.Each update is connected to a clearly defined artifact reconstructed from a human paper.Source: §B.1 · pp. 16–17
  • Training examples provide stage-specific inputs, and each role can search, request targeted readings, and revise its artifact; role outputs use prescribed schemas.The schemas include gap analyses, candidate blocks, reviewer verdicts, and a results-masked report.Source: §B.1 · p. 17, Table 3 · p. 17, Table 4 · p. 17
  • Rubric rewards require structural and citation validity, include reference-free and reference-grounded scores plus citation F1, and apply a protocol-compliance term.A failed validity gate removes rubric reward while leaving the compliance term; reference fields are weighted by role.Source: §B.1 · pp. 17–19, Table 8 · p. 19

B.2 Training Design Choices

  • Metadata-only reading nearly collapsed training, with near-zero mean reward and most rollouts truncated at initialization, so the authors retained full-text reading.They recovered throughput with asynchronous GRPO rather than shorter inputs.Source: §B.2 · p. 19
  • The pairwise innovator reward increasingly favored longer answers and repeatedly collapsed, while the absolute-rubric reward remained stable around 0.66.The authors retained the absolute reference-anchored rubric.Source: §B.2 · p. 19
  • The final reward keeps a small rule-based protocol-compliance deduction, while richer per-turn shaping rewards and privileged-reference rollouts were not adopted.The gap finder’s limit changed from a soft penalty to a hard cap of two gaps per axis; report-writing checks were also refined.Source: §B.2 · pp. 19–20
  • Optimization was tuned for asynchronous GRPO stability using asymmetric probability-ratio clipping, no KL penalty, and generation budgets calibrated to rollout lengths.Higher learning rates, smaller global batches, and different training epochs were also explored during development.Source: §B.2 · p. 20

C Evaluation Metrics and Judge Prompts

The evaluation separates novelty from proposal quality and uses distinct rubrics to assess structural transfer and whether a proposal’s mechanism was independently derivable. Proposal-quality dimensions are scored by LLM judges using proposal-only or problem-conditioned inputs, depending on the dimension.

Novelty measures

  • All and Cutoff rate novelty on a 1–10 scale against closest papers, using the complete comparison corpus or only pre-cutoff papers, respectively.In-domain instead measures the percentage of proposals exhibiting structural cross-setting transfer.Source: C Evaluation Metrics and Judge Prompts · p. 20
  • In-domain novelty is binary: it requires a concrete mechanism transferred across settings through a shared structure, and rejects absent, superficial, or unsupported transfer.Mechanism non-obviousness compares proposals with mechanisms independently derived from pre-cutoff papers before the proposal is revealed.Source: C Evaluation Metrics and Judge Prompts · p. 20

Proposal-quality measures and judge models

  • Proposal quality is scored from 1 to 10 across relevance, clarity, specificity, actionability, soundness, and impact.Clarity, specificity, actionability, soundness, and impact are judged from the proposal alone; relevance also receives the target problem and challenge.Source: C Evaluation Metrics and Judge Prompts · p. 20
  • The primary Qwen3.6-27B judge and four corroborating judges receive the same prompts and inputs.The corroborating judges are Claude-4.8, Gemma-4-31B, GPT-OSS-120B, and GPT-5.4.Source: C Evaluation Metrics and Judge Prompts · p. 20

D Data Construction and Ideation Process Decomposition

The data construction process uses model-generated reference artifacts, disjoint data partitions, and a manual leakage audit. Role-specific adapters are trained and evaluated under configured optimization and inference budgets.

D Data Construction and Ideation Process Decomposition

  • Qwen3.6-27B generated the primary reference artifacts; GPT-5.4 supplied only 76 of 44,931 artifacts after Qwen3.6-27B failed validation after regeneration.The GPT-5.4 contribution was under 0.2% of stage-level artifacts.Source: D Data Construction and Ideation… · p. 25
  • Validation and test partitions are fully deduplicated from training and from each other.Source: D Data Construction and Ideation… · p. 25
  • A manual audit of 50 sampled examples found no problem definitions or challenges exposing target methods through clues such as terminology, assumptions, or examples.The audit compared reconstructed inputs with their source papers.Source: D Data Construction and Ideation… · p. 25

E.1 Training and Inference Configuration

  • Each role has a Qwen3.6-27B LoRA adapter trained with asynchronous GRPO, with role-specific sequence limits and checkpoint selection by held-out validation reward.The selected checkpoints were gap finder step 380, innovator step 310, and report writer step 194; test data was reserved for final evaluation.Source: §E.1 · pp. 25–26
  • Producer roles and the orchestrator run with fixed turn and token budgets, while open-source autoresearch systems and general-purpose agents use their native loops without those imposed caps.Reading is handled by a single-turn paper reader capped at 30,000 input tokens per read; IdeaScientist-Base controls use the same caps as the trained system.Source: §E.1 · p. 26

E.2 Temporal Cutoff and Model-Memory Contamination

The temporal evaluation limits retrieval to pre-cutoff literature, but cannot fully control model-internal memory of later papers. The selected trained checkpoints improved held-out validation reward for each of the three roles.

E.2 Temporal Cutoff and Model-Memory Contamination

  • Retrieval is restricted to papers published before January 1, 2026, but knowledge stored in model parameters cannot be fully controlled.Source: §E.2 · p. 26
  • A title-only probe of 10 post-cutoff papers across five models found no recognized papers, but cannot rule out latent, paraphrased, or method-level memory.The probe comprised 50 model–paper queries and deliberately omitted abstracts.Source: §E.2 · p. 26
  • The January 1, 2026 cutoff is a pragmatic tradeoff, and results indicate performance under controlled retrieval rather than proof of parametric unawareness.All partitions use post-cutoff targets, with the test partition held out from optimization; residual model-memory contamination remains a limitation.Source: §E.2 · p. 26

F.1 Role-Level Validation Reward

  • 0.101 for the gap finder, 0.107 for the innovator, and 0.105 for the report writer are the held-out validation reward gains at selected trained checkpoints over IdeaScientist-Base.Table 11 compares each role’s Base reward with its selected Trained checkpoint; Figure 5 shows training and validation trajectories.Source: §F.1 · p. 27, Table 11 · p. 27, Figure 5 · p. 27

F.2 Training Diagnostics

The selected checkpoint combination improves multiple test-set measures over an earlier combination. Training diagnostics report reward diversity, component-level reward patterns, rollout characteristics, and external-system generation failures.

F.2 Training Diagnostics

  • Across 27,225 gap-finder episodes, median reward was 0.603, only 0.4% of groups had zero variance, and mean within-group standard deviation was 0.104.The reward components show more room for improvement in reference-grounded and citation F1 scores than in the reference-free score; Figure 5 plots per-step training and validation rewards for all three roles.Source: §F.2 · p. 27
  • At the selected gap-finder checkpoint, median rollout length was 17.8K tokens and reward was essentially uncorrelated with sequence length, turns, or papers read.The 512 episodes had median six turns and six distinct papers read; reward correlations were −0.024, −0.031, and −0.017, respectively.Source: §F.2 · p. 27

F.3 Generation Failure Analysis

  • IdeaScientist-Trained and all three open-backbone IdeaScientist-Base runs had no failures among 277 test problems, versus 5 for Claude Code SDK and 8 for Codex SDK.A failure means a test problem lacked a scorable proposal.Source: §F.3 · p. 27

F.4 Checkpoint Selection

  • The selected checkpoints improve valid generation, Citation F1, proposal quality, the reference-grounded score, validity-gate pass rate, and checkpoint-selection average over the earlier combination.Table 12 uses the same primary Qwen3.6-27B judge; its Selection avg. is distinct from Table 1 Overall.Source: §F.4 · p. 28, Table 12 · p. 28

G.1 Test-Time Compute

A controlled sweep finds that relaxing per-role turn budgets improves proposal reward and completion, while the case studies compare human methods, IdeaScientist variants, and external systems across four problems. The case-study methods differ in how they compose safety rules, allocate rollouts, decide when to abstain, and adapt multimodal drafting.

G.1 Test-Time Compute

  • Reward rose from 0.290 to 0.465 and completion from 0.60 to 1.00 as the per-role turn budget increased from low to high on 40 fixed test papers.Mean orchestrator turns consumed rose from 34 to 55, confirming the low-budget setting was compute-starved; other settings were held constant.Source: §G.1 · p. 28, Figure 6 · p. 29

H Judge Reliability

  • Individual-report judge agreement with the leave-one-out consensus ranged from Pearson 0.67 to 0.78, placing all five judges in the same reliable band.System-level agreement ranks systems by each judge’s mean scores for metric families and compares those rankings with the other judges’ leave-one-out mean and the Claude-4.8 judge.Source: H Judge Reliability · p. 28

I Case Studies

  • The case studies compare the human method, IdeaScientist-Base, IdeaScientist-Trained, and seven external systems under shared problem definitions, model settings, inputs, and desired outputs.For the first three methods, tables separate core idea, mechanism, and distinguishing contribution or limitation; external systems are grouped by method family.Source: I Case Studies · pp. 28–29

I.1 Composable Safety Alignment

  • For composable safety alignment, MOSAIC composes input tokens, Base models active-rule interactions with a Set Transformer and FiLM, and Trained projects activations toward jointly satisfying learned rule boundaries.The language model remains frozen across methods; Trained’s linear probe boundaries are behavioral proxies, and nearest feasible projection is not guaranteed.Source: Table 13 · p. 29, §I.1 · pp. 30–31

I.2 Adaptive Rollout Allocation

  • For rollout allocation, Base uses a UCB exploration bonus, while Trained tests whether a promising partial trajectory can be completed to distinguish learnable questions from unreachable ones.The human method jointly resamples and loss-weights questions using confidence and difficulty; Trained discards its diagnostic continuations and its small-sample estimates may be noisy.Source: Table 14 · p. 31, §I.2 · p. 32

I.3 Capability Grounded Subagent Abstention

  • For capability-grounded abstention, the human method learns abstention from examples, Base checks and revises failure explanations, and Trained checks required capabilities against a live inventory before generation.The human method’s reliability depends on data coverage; five baselines use a registry-gate pattern, with AutoResearchClaw adding output auditing and AI-Researcher combining executable constraints with semantic matching and uncertainty.Source: Table 15 · p. 33, §I.3 · p. 34

I.4 Adaptive Multimodal Speculative Decoding

  • For multimodal speculative decoding, Base gates visual-feature injection using divergence between text-only and vision-conditioned drafts, while Trained routes each turn using the target model’s prefill-stage visual attention.The human method updates an ensemble from target verification; Base’s pseudocode does not clearly connect its gate to the final draft-token distribution, while Trained’s attention is a proxy and prefill is not zero-latency.Source: Table 16 · p. 34, §I.4 · pp. 35–36
Loading 2610.04074v1…