PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

Qihao Zhao, Yangyu Huang, Yalun Dai, Lingao Xiao, Jianjun Gao, Xin Zhang, Wenshan Wu, Scarlett Li, Yang He, Yan Lu, Yap Kim Hui

arXiv:2607.04439v1cs.AI

TL;DR

Research ideation needs reusable ways to ground evidence, identify bottlenecks, and audit proposed directions against prior art. ResearchStudio-Idea provides a three-skill suite, centered on IdeaSpark, that turns outcome-grounded patterns into auditable idea cards. Its corpus analysis induces 15 ideation patterns and 31 sub-patterns for literature-grounded ML ideation.

  • Problem

    LLM ideation systems still lack a reusable skill layer that connects research evidence and bottlenecks to auditable candidate directions before experiments begin.

  • Method

    ResearchStudio-Idea combines literature search, prior-art collision checking, and IdeaSpark’s outcome-grounded pattern guidance to produce one auditable idea card.

  • Results

    The study analyzes 1,947 ICLR, ICML, and NeurIPS papers from 2021–2025 to induce 15 ideation patterns, 31 sub-patterns, and 28 research domains.

  • Takeaways & Limitations

    The resulting pattern library makes outcome-grounded ideation patterns executable at inference time for literature-grounded ML research ideation.

  • Takeaways & Limitations

    The study’s corpus and evaluation are bounded to three ML conferences, so its patterns and notion of strong directions may not transfer to other scientific communities.

Abstract

from arXiv · show

Large language models have made research ideation increasingly accessible, yet effective idea development requires more than generating candidate directions. Researchers must ground a problem in current literature, identify meaningful bottlenecks, differentiate from existing solutions, and evaluate risks before committing to implementation. We present ResearchStudio-Idea as a reusable skill suite for this first mile of research ideation. The suite includes Paper-Search, a standalone multi-source literature search skill; Scoop-Check, a standalone prior-art collision checker for novelty claims; and IdeaSpark, the end-to-end skill that composes evidence grounding, pattern-guided generation, collision retrieval, audit, and idea-card rendering into one workflow. IdeaSpark is constructed from a corpus of 1,947 machine learning conference papers collected from ICLR, ICML, and NeurIPS between 2021 and 2025, including Oral papers, a separately tracked high-citation subset, and rejected submissions. Analysis of these outcomes reveals 31 recurring ideation sub-patterns, consolidated into 15 reusable ideation patterns. Each pattern is operationalized as a structured card containing research contexts, bottleneck types, differentiation strategies, supporting precedents, and common failure modes. Given a research problem and an evidence bundle, IdeaSpark evaluates evidence readiness, reconstructs the surrounding research context, identifies unresolved bottlenecks, selects relevant patterns, instantiates one candidate direction, retrieves potentially conflicting prior work, and performs outcome-informed auditing. This workflow transforms reusable ideation patterns into traceable research proposals. Blind automated-judge evaluations show that IdeaSpark consistently produces stronger research proposals than no-skill and generic-skill baselines while maintaining competitive novelty.

10 Do Rejected Papers Inhabit a Different Strategy Space? · 1 Introduction

ResearchStudio-Idea addresses the missing middle between literature evidence and actionable, auditable research ideas through three reusable skills and an outcome-grounded pattern map. Its rejected-paper analysis finds that rejected work shares the accepted papers’ high-level strategy space, making it useful chiefly for contrastive failure analysis rather than as a separate strategy class.

  • 1.1 Problem and scope: LLM research systems can retrieve literature and generate proposals, but early-stage ideation still requires grounding bottlenecks, unresolved opportunities, differentiation, and risks.The report frames this as the first-mile problem: producing plausible proposals is easier than developing evidence-grounded, distinguishable research directions.
  • 1.1 Problem and scope: ResearchStudio-Idea packages this missing layer into Paper-Search, Scoop-Check, and IdeaSpark for literature grounding, prior-art collision checking, and end-to-end idea generation and audit.Scoop-Check decomposes novelty claims into problem framing, core mechanism, key insight, and application domain before comparing retrieved prior work.
  • 1.1 Problem and scope: IdeaSpark is an inference-time skill that combines pattern cards, workflow prompts, schemas, retrieval hooks, and validators rather than serving as an acceptance predictor, novelty score, or paper-writing agent.Its empirical role is to connect search, generation, and review functions into one outcome-grounded ideation workflow.
  • 1.1 Problem and scope: The report analyzes 1,947 ICLR, ICML, and NeurIPS papers and evaluates IdeaSpark-generated ideas with blind automated judges.The analysis mines conference outcomes into reusable patterns, while the evaluation tests the composed skill rather than claiming human-validated acceptance prediction.
  • 1.2 Motivation: The missing middle connects retrieved evidence and identified bottlenecks to reusable ideation patterns and auditable candidate research directions.Patterns encode how papers connect gaps to methods, assemble differentiation evidence, and expose failure modes before experiments begin.
  • 1.4 Main empirical findings: The pipeline induces 15 higher-level ideation patterns from 31 fine-grained sub-patterns, with 15 reflecting the current operational granularity rather than a canonical ML innovation ontology.The induction comes from the current extraction, abstraction, embedding, clustering, and induction pipeline, and stability across prompts, seeds, and models remains future work.
  • 10 Do Rejected Papers Inhabit a Different Strategy Space?: Rejected and accepted papers often share the same high-level pattern space: reject-only re-clustering maps every rejected-paper cluster onto the existing 15-pattern vocabulary.Rejected submissions therefore provide contrastive evidence about weak instantiations, failure modes, and boundary cases rather than a separate negative strategy class.
  • 1.4 Main empirical findings: Papers commonly combine multiple ideation patterns, with k = 2 modal across Oral, HC, and Reject papers and a 33.6% tail at k ≥3.This supports generating one to three pattern roles, while domain statistics remain audit context because acceptance and impact vary by domain-pattern cell.

2 Related Work · 3 Dataset Construction

Related work spans autonomous research agents, search-based ideation, pattern and knowledge-graph induction, and idea evaluation, while IdeaSpark combines these ingredients through contrastive pattern cards and retrieval-grounded auditing. Its dataset contains 1,947 deduplicated ICLR, ICML, and NeurIPS papers from 2021–2025, labeled by Oral, High-Cited, and Reject outcomes with uneven review metadata coverage.

  • 2.1 End-to-end “AI scientist” systems: End-to-end systems automate research lifecycles, but documented failures include training-default bias, implementation drift, context degradation, premature success claims, thin domain knowledge, and weak experimental taste.AI Scientist, AI-Researcher, Agent Laboratory, and related systems represent this family, whose autonomy remains fragile.
  • 2.2 Multi-agent and search-based ideation: Multi-agent and search-based methods target ideation through role-separated collaboration, concept-network exploration, iterative planning, external retrieval, and literature-guided search evolution.Examples include VirSci, IRIS, Deep Ideation, Nova, FlowPIE, and benchmarked reflection, evolution, recursive-decomposition, and multimodal workflows.
  • 2.3 Pattern induction from conference outcomes: Pattern- and knowledge-graph methods induce reusable structure from accepted papers or large literature graphs, supporting controllable generation, motivational reasoning, positioning, and constrained hypothesis formation.Prior systems include Oral-paper taxonomies, MoRI, MotivGraph-SoIQ, Navigating Ideation Space, SciMuse, and KGgrounded hypothesis generation.
  • 2.4 Novelty, evaluation, and benchmarks: Evaluation work separately benchmarks novelty collision, idea quality, and scientific discovery, finding substantial expert–LLM disagreement and lower executed performance despite higher perceived novelty.NovBench contains 1,684 paper-review pairs, RINoBench 1,381 expert-judged ideas, and AI Idea Bench 2025 curates 3,495 papers.
  • 2.6 How IdeaSpark differs from current idea-generation methods: IdeaSpark combines pattern induction, accept/reject contrast, and retrieval-grounded generation, using one non-parametric data-induced pattern card both to select an ideation move and audit its candidate.Unlike learned candidate ranking or parametric motivation-to-method policies, the workflow stops at one reviewer-facing idea card and explicitly incorporates failure signals.
  • 3.1 Scope and labeling: The corpus collects ICLR, ICML, and NeurIPS papers from 2021–2025 through OpenReview and Semantic Scholar, retaining decisions, reviews, meta-reviews, identifiers, bibliographic fields, and citation counts.Oral represents program-committee preference, High-Cited community adoption, and Reject a concrete failure signal; labels may overlap, including 49 Oral–High-Cited papers.
  • 3.2 Metadata coverage: 1,947 unique papers remain after deduplication, while only 340 carry a meta-review and 56 are excluded from the embedding pool because abstracted fields are incomplete.The label-inclusive total is 1,996 because 49 papers overlap between Oral and High-Cited; review-derived fields are treated as low-evidence auxiliary data.
  • 3.3 Convention for in-paper paper references: Corpus papers use venue-year identifiers resolving to OpenReview submissions, while approximately 1,088 papers are cited in the report and substantive body-text discussions receive bibliography entries.The prepared dataset serves as the authoritative bibliographic resolution layer without listing the complete corpus bibliography.

4 Two-Stage Innovation-Signature Extraction · 5 Unsupervised Pattern Discovery

The pipeline extracts structured innovation signatures, rewrites strategy-bearing fields into domain-agnostic form, and clusters them to discover reusable ideation sub-patterns. Applied to 1,947 papers, this process yields 31 sub-patterns whose acceptance compositions provide descriptive risk signals rather than recommendations.

  • 4.1 Stage 1: Eight base fields: Each paper receives eight base fields covering its innovation strategy, key reasoning step, non-obviousness, trigger condition, review feedback, acceptance signal, and contribution type.The first four fields describe strategy, while the latter four preserve review and outcome evidence for later card construction.
  • 4.1 Stage 1: Eight base fields: All 1,947 papers pass Stage 1 extraction under a JSON schema requiring non-empty review-praise and reviewer-concern lists of length 3.Stage 2 rewrites only the four strategy-bearing fields; the remaining fields supply review and outcome evidence.
  • 4.2 Stage 2: Four domain-agnostic rewrites: Stage 2 replaces domain-specific descriptions with four strategy-only fields, using generic placeholders and imperative phrasing to support strategy-level rather than topic-level clustering.The rewritten fields are abstract strategy, abstract key step, abstract why non obvious, and abstract trigger condition.
  • 5.1 Embedding: The four abstract fields are concatenated into records averaging 1,185 characters, embedded with text-embedding-3-large at 3,072 dimensions, and L2-normalized.UMAP subsequently reduces the embedding space to 10 dimensions before clustering.
  • 5.2 Clustering: 31 ideation sub-patterns emerge at min cluster size=10, achieving silhouette 0.584 while leaving 902/1,891 = 47.7% of papers unclustered.The unclustered label reflects embedding geometry between density peaks rather than weak or strategy-empty papers.
  • 5.3 Cluster inventory: The 31-cluster inventory ranges from 13 to 86 papers per cluster, with mean 31.9 and median 22, and is labeled and expanded into tactic-level disambiguation cards.The cards encode structural recipes, sibling differentiation, selection conditions, and failure modes for subsequent pattern induction.
  • 5.4 Per-cluster acceptance composition: Six clusters are Oral-safe, one is Reject-warn, and the remaining 24 are mixed based on cluster-level Oral, HC, and Reject compositions.These flags are informational risk statistics sensitive to sample size, year, conference, topic saturation, and execution, not recommendation labels.

6 Ideation-Pattern Induction

The study induces a reusable 15-pattern taxonomy from 31 fine-grained ideation clusters, with each pattern specified by a definition, operational signature, and application condition. The framework covers the corpus completely at paper level, while multi-label analysis shows that two-pattern compositions are most common and three-or-more combinations form a substantial tail.

  • Taxonomy induction: Claude Opus 4.7 induced a 15-pattern taxonomy from 31 clusters under constraints requiring reusable reasoning strategies and structured operational definitions.The taxonomy and cluster mapping were returned in one structured call without edits, and each pattern is documented in a per-pattern card.
  • Coverage: 100% of clustered papers received a primary pattern, with 989 of 989 clustered papers covered and every cluster mapped to exactly one pattern.Cluster-level inheritance covered 52% of the 1,891-paper embedding pool; the remaining papers were handled by paper-level multi-label tagging.
  • Pattern distribution: The taxonomy has a long tail: the top three patterns account for 384 papers, whereas the bottom three account for 48 papers.The largest patterns are Audit and Pivot an Assumption (n=181), Substitute the Operator or Representation (n=109), and Liberate a Fixed Generative Component (n=94).
  • Pattern composition: Among clustered papers, 729 (73.7%) carry both a primary and non-trivial secondary pattern, making multi-strategy assignment the norm.The remaining 260 (26.3%) carry only a primary; the 902 unclustered papers were assigned later through multi-label tagging.
  • Multi-label analysis: 59.2% of papers execute exactly two ideation patterns, while 33.6% execute three or more and 7.2% execute a single pattern.The k=2 mode is shared across Oral, HC, and Reject, and class means cluster around k̄ = 2.3.

7 Acceptance and Impact Analysis

Acceptance differences between ideation patterns are small on the Oral-vs-Reject axis, while the PC-vs-community axis provides a stronger discriminating signal. The analysis therefore treats execution and structural fit as more informative than main-pattern choice, using these statistics only for audit context.

  • Analysis scope: Cluster-level analysis covers 482 of 1,014 Oral papers and 367 of 722 Reject papers, with denominators NO = 482, NHC = 172, and NR = 367.Unclustered papers are covered by multi-label tagging in §6.5.
  • Acceptance bias: The Oral-vs-Reject bias ∆OR spans only ±2.9 pp, indicating that main-pattern choice explains little acceptance variance.Reframe as a Solvable Object reaches +2.9, while Decompose for Differentiated Treatment reaches −2.9.
  • Pattern observations: Audit and Pivot has the largest cluster-level shares in both Oral and Reject papers, with 19.5% Oral, 21.5% Reject, and ∆OR=−2.0 pp.Its ∆OH=+13.1 pp is the largest positive PC bias, separating 94 Oral from 79 Reject papers through execution quality.
  • Pattern observations: Cross-modal-alignment has ∆OH=−11.2 pp, reflecting a 17.4% HC share versus a 6.2% Oral share and weaker PC elevation.The passage characterizes these papers as reproduced and cited heavily, but not elevated to Oral at the same rate.
  • Pattern observations: Reframe as a Solvable Object is the clearest Oral signal, combining ∆OR=+2.9 pp, ∆OH=+7.1 pp, and cosine 0.956 with Audit and Pivot.The resulting recipe is to recast an intractable formulation as solvable, then audit and pivot the recast’s load-bearing assumption.
  • Use in the skill: IdeaSpark uses neither acceptance nor PC-vs-community statistics as selection priors; pattern selection follows structural fit, while statistics enter only during audit.The PC-vs-community axis separately captures disagreement between committee and citing-community judgments.

8 Domain Distribution and Ideation-Pattern Breadth

The analysis induces 28 research domains and shows that ideation-pattern success varies substantially by domain, with one limit-surpassing pattern providing the clearest cross-domain Oral signal. Most patterns remain broad enough to support domain-agnostic use, while narrower patterns reflect domain-specific operational substrates rather than uniformly poor outcomes.

  • Domain induction: 28 induced domains cover 98.5% of papers, with each paper assigned an average of 1.81 domains.The taxonomy was induced without manual edits; domains range from 256 to 16 papers.
  • Cross-domain outcomes: The 28 × 15 grid contains 392 populated cells, but Oral rates are heterogeneous because the same pattern performs differently across domains.Of 420 cells, 292 have nO + nR ≥5; rates are masked below that threshold.
  • Cross-domain outcomes: Characterize a Limit, Then Surpass It occupies four of the five strongest cells, including 100% Oral in Trustworthy & Responsible ML and Learning Theory & Generalization.The Trustworthy & Responsible ML cell has nO=12, nR=0, while Learning Theory & Generalization has 11/0.
  • Cross-domain outcomes: The weakest dense cells reach only 9%–25% Oral, confirming that success conditions do not transfer mechanically across domains.Examples include Distributed & Federated Learning × Reframe as a Solvable Object at 9% and Graph Learning × Substitute the Operator or Representation at 21%.
  • Ideation-pattern breadth: Diagnostic and framing patterns are broadest, while construction-level patterns reach 22–25 domains and several domain-narrow patterns remain Oral-enriched.Audit and Pivot an Assumption and Reframe as a Solvable Object reach all 28 domains; Decompose for Differentiated Treatment reaches 27, whereas Characterize a Limit, Then Surpass It reaches 12.

9 Temporal and Conference Structure

From 2021–2025, ideation patterns show clear temporal risers and venue-specific signatures. These descriptive trends inform audit context rather than generation priors, while early-year imbalance limits interpretation of small changes.

  • Temporal change: Decompose and Delegate to Solvers rose fastest, from 0.0% early to 6.3% late (∆= +6.3 pp), alongside growth in LLM-driven solver pipelines.The early period averages 2021–2022, and the late period averages 2024–2025.
  • Temporal change: Design a Confound-Isolating Diagnostic increased from 11.3% to 15.6% (∆= +4.3 pp), reflecting controlled-perturbation evaluation becoming a first-class research move.Unify Heterogeneous Inputs into One Space was the third clearly rising pattern, with ∆= +2.6 pp.
  • Temporal change: Early-period shares aggregate roughly 150 papers, and changes with |∆| < 2 pp are not statistically robust; identified movers all have ∆≥2.5 pp.The 2021 and 2022 slices contain 77 and 73 multilabeled papers, respectively, versus 537–606 papers per year from 2023–2025.
  • Venue signatures: ICML over-indexes on assumption audits, with Audit and Pivot an Assumption at 56.2% versus 52.4% elsewhere (+3.8 pp), and limit characterization at 5.7% versus 4.3%.These patterns align with sharply bounded analytical contributions.
  • Venue signatures: ICLR favors construction, while NeurIPS favors decomposition and invented supervision: Encode Structure by Construction reaches 20.6% versus 16.2%, and Decompose for Differentiated Treatment reaches 31.4% versus 26.8%.ICLR also over-indexes on Liberate a Fixed Generative Component at 16.2% versus 13.3%; NeurIPS’s Manufacture the Supervisory Signal reaches 18.0% versus 14.3%.

10 Do Rejected Papers Inhabit a Different Strategy Space? · 11 Ablation: Embedding Model and Abstraction Stage

Rejected papers occupy the same 15-pattern strategy taxonomy as accepted work, though their fine-grained distribution shifts toward audit and unify patterns. Ablations show that the production configuration is preferred for construct validity and stability: paraphrase-strong embeddings preserve strategy similarity, while domain-agnostic rewrites prevent topic-driven collapse.

  • 10 Do Rejected Papers Inhabit a Different Strategy Space?: Rejected work clusters within the existing 15-pattern taxonomy rather than occupying a separate strategy space.Every Reject-only cluster maps to an existing pattern, with no out-of-taxonomy label.
  • 10.1 Reject-only clustering and mapping: 13 Reject-only clusters formed from 711 papers, with 40.5% unclustered; all mapped to existing main-pattern centroids at 0.897–0.986 cosine.Eleven of the 15 main patterns received at least one Reject-only cluster.
  • 10.2 Interpretation: 10 of 15 patterns had comparable Reject counts across clustering procedures, with |∆| ≤10, indicating a stable strategic shape.Reject-only and main-taxonomy assignments therefore broadly agree despite separate versus joint clustering.
  • 10.2 Interpretation: +52 papers went to Unify Heterogeneous Inputs into One Space and +19 to Audit and Pivot an Assumption, while constructive patterns received fewer.The divergences were −23 for Encode Structure by Construction, −15 for Reframe as a Solvable Object, and −10 for Manufacture the Supervisory Signal.
  • 11.1 Embedding model: OpenAI vs SPECTER2: 31 vs. 23 clusters and 0.584 vs. 0.438 silhouette show that text-embedding-3-large outperformed SPECTER2 at min cluster size = 10.SPECTER2’s silhouette declined to 0.333 at min cluster size = 25, with 41–54% unclustered.
  • 11.1 Embedding model: OpenAI vs SPECTER2: The general-purpose paraphrase encoder better preserves strategy-equivalence relations than SPECTER2’s citation-informed, topic-oriented representation.The target similarity is whether papers re-instantiate a proven mechanism at appropriate domain-specific granularity, not merely whether they are topically adjacent.
  • 11.2 Abstraction stage: silhouette is not enough: 0.586 vs. 0.584 silhouette and 35.7% vs. 47.7% unclustered initially favored base fields over production at min cluster size = 10.This raw advantage was misleading because the base-fields variant became unstable across the sweep.
  • 11.2 Abstraction stage: silhouette is not enough: k = 2 mega-clusters with silhouette 0.884 and 0% unclustered revealed topic-driven collapse for base fields, whereas production retained k between 12 and 19 and silhouette 0.53–0.58.Domain-agnostic rewrites remove topic anchors, preserve strategy signal, and remain stable across coarser HDBSCAN settings.

12 Discussion · 13 IdeaSpark Design

The Discussion identifies composition, execution, audit, and domain-conditioned effects as the main empirical lessons, while IdeaSpark operationalizes them as a literature-grounded workflow that generates and audits one reviewer-facing proposal. Its retrieval gates, provenance discipline, validators, and bounded revision path constrain unsupported reasoning, but novelty, human-rated quality, and held-out rediscovery remain open.

  • 12.1 Empirical takeaways: The 15-pattern strategy space is shared across accepted and rejected papers, so execution, composition, and failure-mode handling—not pattern choice alone—distinguish outcomes.Reject-only clusters reproduce the same vocabulary but drift toward audit/unify lanes; composition content matters more than size, with k=2 as the empirical mode.
  • 12.1 Empirical takeaways · 13.2 Design principles: The taxonomy serves as standalone diagnostic vocabulary: researchers can compare ideas against success conditions, while reviewers can identify specific failure modes.The skill operationalizes these uses inside an LLM scaffold rather than treating patterns as contribution claims.
  • 13.1 Positioning and two-tier architecture · 13.4 Output surface: IdeaSpark converts an under-specified direction into one compact idea-card bundle, or stops with a grounding diagnosis or failed-audit report.The card includes motivation, method flow, core claim, falsification prediction, and feasibility validation; experiment planning and empirical engineering remain the user’s responsibility.
  • 13.1 Positioning and two-tier architecture: The two-tier architecture separates compact runtime orchestration and validators from progressively read evidence assets, including 15 pattern cards, 31 sub-pattern cards, matrices, saturation records, and failure modes.Phase 2 reads only the definitions and tactics needed for candidate instantiation, while later phases use broader evidence for auditing.
  • 13.2 Design principles · 13.3.3 Phase 2: Pattern-guided ideation: The workflow follows literature first, vocabulary second, idea third: retrieved papers establish a structural bottleneck, patterns guide a research move, and sub-patterns guide concrete mechanism instantiation.Pattern frequency and saturation inform audit context rather than generation priors, preserving generation freedom while anchoring evaluation.
  • 13.3.1 Phase 0: Literature grounding · 13.5 Faithfulness: resisting hallucination: Phase 0 combines four complementary sources, non-overlapping 0–6-month and 6–24-month windows, deduplication, meaning-based recall, and open-access full text to ground downstream checks.Phase 1 hard-gates on a full-text cache, and every bottleneck, citation, and collision check is intended to trace to retrieved records rather than model memory.
  • 13.3.2 Phase 1: Bottleneck identification: Phase 1 builds a method-lineage tree to identify additive and subtractive gaps and prevent regression to fixes already implemented by ancestors.It proceeds only with a concrete literature-grounded bottleneck; unsupported or overly broad directions stop with a diagnostic, while some benchmark and infrastructure directions remain uncovered triggers.
  • 13.3.3 Phase 2: Pattern-guided ideation · 13.3.4 Phase 3: Quality gauntlet · 13.3 Workflow and phase contracts: Phase 2 generates candidates containing a mechanism, gap-closure rationale, differentiation, compute budget, and falsification prediction, while Phase 3 performs evidence-backed recipe, collision, and failure-mode checks.The hard-floor layer abandons candidates matching clear reject lessons, unmitigated reject-enriched compositions, or exact-mechanism overlap; borderline cases may be revised once.

14 Evaluation: Generated-Idea Quality and Novelty

Blind automated evaluation on 100 forward-held-out problem seeds finds that IdeaSpark achieves substantially higher idea quality than the baselines while remaining competitively novel. The results also show that novelty alone can reward vague, empty ideas, so quality and novelty must be considered jointly.

  • Evaluation setup: All sources are normalized to matched three-section formats and comparable length before blind judging, with formal equations required in every Method.Motivation is constrained to [259, 330] words and Method to [449, 866] words; IdeaSpark author-decision annotations are stripped before scoring.
  • Quality: IdeaSpark scores 3.87/4 on quality and ranks first on 88 of 100 seeds, while no other system exceeds 2.6.Opus-self-gen scores 2.57 and bare Opus-4.8 scores 2.56, showing that a structured skill alone does not lift quality.
  • Quality: IdeaSpark’s quality gain is attributed to corpus-grounded pattern cards and a multi-phase, audit-anchored workflow rather than model family or retrieval alone.IdeaSpark, Opus-self-gen, and Opus4.8 (bare) share one backbone, while Opus-self-gen also has live retrieval.
  • Novelty: GPT-5.5 bare reaches 3.73 novelty with 214/300 judgments at L4 but scores 1.00 quality with zero wins, exemplifying the “novel-but-empty” regime.The skill-based systems occupy the high-quality region at L3 medium overlap, where shared framing and domain coexist with a distinct mechanism.

15 Limitations

The study’s claims remain scoped to the idea stage, with limitations in corpus coverage, automated endpoint evaluation, and interpretation of judge metrics. Each limitation is paired with a concrete next step or bounded reading of the results.

  • Scope: These limitations do not change the paper’s claims, which remain scoped to the idea stage and identify where additional evidence would extend them.Each limitation is presented as a bounded property of the current study paired with a concrete next step.
  • Coverage: The corpus covers only ICLR, ICML, and NeurIPS outcomes and public reviews, so its induced patterns reflect mainstream ML-conference research rather than science generally.Evaluation seeds are also entirely ICLR 2026 Orals, leaving under-represented fields and venues outside the corpus distribution.
  • Evaluation: Both evaluation axes use LLM-based skills, measuring agreement with structured automated reviewers rather than human program-committee judgment.A blind expert study is planned as the next stage before making any acceptance-level claim.
  • Metric interpretation: Idea-quality rewards novel mechanisms, so benchmark, systems, and pure-measurement contributions score lower on method depth by construction.Quality is therefore interpreted relatively among generation systems, not as an absolute measure of research value.
  • Metric interpretation: Novelty is meaningful only jointly with quality because sufficiently generic ideas can evade prior-art collision and appear novel.The passage identifies the GPT-5.5 case as an example of this failure mode.

16 Artifact Cards · 17 Responsible Use · 18 Conclusion

ResearchStudio-Idea is presented as a bounded, auditable suite for literature-grounded ML ideation, with explicit data, model, applicability, and responsible-use constraints. Its conclusion emphasizes reusable strategic operators from public conference outcomes rather than acceptance prediction, paper writing, or replacement of experimental judgment.

  • 16.1 Data card: The corpus uses public metadata and reviews, while author identities remain metadata only and are excluded from pattern induction and recommendation.
  • 16.2 Model and backend card: The pipeline uses LLMs for extraction, taxonomy induction, card generation, and skill phases, plus an embedding model for clustering.Principal models include Claude Sonnet 4.6, Claude Opus 4.7, OpenAI text-embedding-3-large, and Claude Opus 4.8 high.
  • 16.3 Skill card: IdeaSpark is intended for literature-grounded ML directions with a stated domain, adjacent literature, constraints, and desired contribution type, not confidential literature or broad agenda-setting.It returns a structured idea card rather than supporting pure benchmark construction, complete paper writing, automated submission production, or claims requiring private peer-review data.
  • 17 Responsible Use: IdeaSpark is scoped as an ideation scaffold rather than a paper-writing agent, producing one idea card with failure paths and audits against reject lessons and prior-art threats.The passage frames plausible-looking proposal generation as creating risks including low-quality proposals, reviewer burden, shallow submissions, and overconfident novelty claims.
  • 17 Responsible Use: Rejected papers and reviews are used only as aggregate pattern-level evidence about failed strategic moves, never to rank, profile, or single out papers, authors, or reviewers.Recommended public-deployment safeguards include rate limits, artifact provenance, citation grounding, and warnings that the system does not predict acceptance.
  • 18 Conclusion: ResearchStudio-Idea comprises Paper-Search, Scoop-Check, and IdeaSpark, supported by a 1,947-paper analysis of ICLR, ICML, and NeurIPS outcomes from 2021–2025.The analysis induced 15 ideation patterns and 31 recurring sub-patterns, while IdeaSpark combines grounding, collision checking, pattern guidance, and auditable idea-card production.
  • 18 Conclusion: The paper claims that public conference outcomes contain reusable strategic operators whose success and failure conditions can be packaged into practical skills.It explicitly does not claim that an LLM can generate Oral papers, predict acceptance, or replace experimental judgment; the automated-judge evaluation provides initial endpoint evidence.

Appendix · A End-to-end Generated Idea Card

The appendix presents real IdeaSpark artifacts, including a rank-1 held-out ICLR 2026 idea card and its evidence trail. The card proposes Phase-Stratified GRPO, which redistributes verifier-free credit across answer phases while preserving the original GRPO objective.

  • Appendix: The appendix reproduces a generated idea card, its JSON evidence trail, a reject-path run, and a pattern/sub-pattern card as real §14 artifacts.The held-out ICLR 2026 seed concerns reinforcement learning for long-context reasoning; its evidence trail includes the core claim, falsification, and literature deltas.
  • A End-to-end Generated Idea Card: The proposed method, Phase-Stratified GRPO, targets long-context reasoning by assigning credit to plan, retrieve, reason, and recheck phases rather than broadcasting one answer-level score.The card frames LoongRL with GRPO as the current substrate and identifies uniform answer-level credit as the motivation for redistribution.
  • A End-to-end Generated Idea Card: The evaluation trains matched plain-GRPO and phase-stratified models, then compares accuracy at 16,000, 32,000, 64,000, and 128,000 tokens on RULER and multi-hop benchmarks.The extended multi-hop suite includes HotpotQA, MuSiQue, and 2WikiMultiHopQA, with decisive-phase concentration measured alongside accuracy.
  • A End-to-end Generated Idea Card: The per-phase credit density combines divergence contrast with standardized model confidence using β1 = 1.0 and β2 = 0.3, then applies a phase-wise softmax.Confidence is computed from already available token log-probabilities, avoiding an extra forward pass; the resulting density sums to 1 across phases.
  • A End-to-end Generated Idea Card: The method splits answers into plan, retrieve, reason, and recheck phases using boundary markers, falling back to fixed 512-token chunks when markers are absent.The splitter is rule-based and processes each generated answer once without adding a learned scorer.
  • A End-to-end Generated Idea Card: Within each group of 16 attempts, correct and incorrect answers are paired by their longest shared phase-type prefix, and the first divergence receives +1 for the correct attempt and −1 for the wrong attempt.Tie-breaking uses the lowest attempt id, making the pairing reproducible.
  • A End-to-end Generated Idea Card: Training replaces uniform token-level advantages with Ag,s = Ag ∗(Sg ∗dg,s), preserving the original average advantage while concentrating credit on higher-density phases.The remaining GRPO probability-ratio and clipping update is unchanged, so the method moves rather than creates or destroys total credit.

B Kill-switch and Falsification Fields … 2. gap closure reject check — triggered

The pipeline combines mechanism-aware falsification and kill-switch fields with a corpus-anchored reject path that abandons ideas when exact prior-art overlap creates an unaddressable hard floor. In the showcased run, the candidate’s novelty residual was only an internal proof quantity, so gap-closure checks rejected it rather than treating recipe application as sufficient novelty.

  • B Kill-switch and Falsification Fields: Mechanism-aware falsification uses a named load-bearing variable, a non-tautological negative control, and byte-identical kill-switch preservation across phases.These fields support reviewer-defensible auditing but are deliberately omitted from the rendered idea card.
  • B Kill-switch and Falsification Fields: The candidate claims that phase-stratified, verifier-free credit redistribution recovers long-context learning signals diluted by uniform broadcast at 128K.The proposal redistributes GRPO’s single trajectory-level advantage onto per-phase credit density without adding a separate reward model or critic.
  • B Kill-switch and Falsification Fields: The minimal falsification test compares vanilla and phase-stratified GRPO from identical bases and compute across 16K, 32K, 64K, and 128K evaluations.The predicted accuracy gap should increase with context length, with stratified credit roughly matching vanilla at 16K but improving materially at 128K.
  • B Kill-switch and Falsification Fields: The named mechanism variable is concentration ratio C, while phase-label permutation holds C fixed but randomizes placement and should collapse 128K accuracy to vanilla levels.This negative control tests whether gains arise from concentrating credit on contrast- or self-score-identified pivotal phases rather than from concentration alone.
  • C Reject-path Output (Phase 3 Abandon): When Phase 3 encounters an unaddressable hard floor, the system emits failed.md instead of an idea, abandoning candidates whose exact mechanism already appears in the literature.The showcased seed concerned non-asymptotic analysis of sticky Track-and-Stop, and the generated certificate was marked abandon.
  • 1. paper pointed threat — exact-mechanism collision (hard floor, unaddressable): The threat paper already provides non-asymptotic guarantees for Track-and-Stop and Sticky Track-and-Stop, exactly matching the candidate’s headline deliverable.The candidate’s additive B(mu) burn-in functional was judged an internal proof quantity that any such bound must control, not an external differentiator.
  • 2. gap closure reject check — triggered: C19 and C29 both fired because the candidate mechanically re-derived an already established guarantee while holding the sticky-Track-and-Stop algorithm fixed.The recipe was applied correctly and no anti-pattern matched, making prior-work overlap—not recipe misuse—the failure axis.
  • 2. gap closure reject check — triggered: The root cause was that the seed direction essentially duplicated the title of an existing 2025 paper, leaving only an internal B(mu) lemma rather than an externally differentiating claim.The harness later rerouted the run to another gap for scoring, while preserving this artifact as the genuine abandon-branch output.

User-side options

The current Track-and-Stop direction should be dropped or replaced because its non-asymptotic analysis is already covered by a 2025 paper. Viable alternatives include reframing around uncovered finite-confidence or lower-bound questions and rerunning the critique with the threat paper included.

  • Collision assessment: Drop or replace the direction: non-asymptotic analysis of sticky Track-and-Stop is already occupied by the 2025 paper arXiv:2505.22475.The passage identifies this prior work as a direct collision with the proposed headline.
  • Alternative directions: Re-frame around an uncovered residual, such as finite-confidence tradeoffs for forced exploration or instance-dependent lower bounds on burn-in.The suggested pivots contrast a deferred finite-delta analysis and an impossibility result with an upper bound already supplied by the prior paper.
  • Next step: Re-run Phases 2.1 and 2.2 on a different selected gap from the four gaps in phase1 output.json, then re-critique with the threat paper in scope.This workflow is proposed to reassess remaining candidates after incorporating the newly identified prior work.

Artifacts on disk

The workflow produced literature, selection, generation, collision, and critique artifacts on disk. The candidate passed citation gating but was abandoned after 173 collision hits and a hard-floor critique, so no phase4 evaluation was produced.

  • phase0/ stores the literature table, results, and full-text cache.
  • phase2 outputs record selection and a generated candidate that passed the citation gate.
  • 173 collision hits were recorded, with arXiv:2505.22475 marked critical.
  • The critique verdict was abandon at the hard floor, leaving no phase4 or evaluation/opus4 8/85 output.The failure record is stored in phase 3 failed.md.

D Exemplar Ideation Pattern and Sub-pattern Card

The Audit and Pivot an Assumption pattern identifies a load-bearing assumption, then relaxes or violates it to produce a genuinely new achievable result. Strong executions empirically validate the pivot, certify tightness, explain prior failures, and differentiate sharply from nearby precedents while avoiding incremental or weakly justified combinations.

  • Pattern definition: The exemplar audits a load-bearing implicit assumption and pivots by relaxing it to a weaker condition or violating it to unlock new behavior.The pivot must change what is achievable rather than cosmetically restate the prior result.
  • Validation criteria: A valuable pivot demonstrates that the relaxed or violated condition holds in the real setting and pairs the resulting bound with a tightness certificate.Evidence may come from natural data or a bounded adversary; matching lower bounds, exact equivalence, or impossibility results establish that the audited assumption was the genuine barrier.
  • Validation criteria: Prior methods must fail specifically because of the audited assumption, establishing that the pivot addresses a binding constraint rather than an incidental simplification.This failure-case characterization should precede the positive result.
  • Failure modes: Common failures include producing no new achievable result, stacking components without isolating the load-bearing pivot, or failing to distinguish a near-identical prior pivot.These shortcomings make the contribution appear incremental, engineering-driven, or redundant.
  • Sub-pattern instantiation: The security/detection sub-patterns relocate signals across loci, require mechanistic explanations, and close against informed adversaries rather than relying on fixed-baseline demonstrations.Examples include supervision or label space, intermediate-representation geometry, distribution shape, deterministic constructions, exploitable security properties, and structural pipeline root causes.
Loading 2607.04439v1…