PaperlayerView exact revision on arXiv

Source-linked AI summary

Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Yifan Zhou, Qihao Yang, Yan Li, Donggang Li, Xiru Hu, Hokin Deng, Ziyang Gong, Xuanyi Zhou, Huacan Wang, Xiangchao Yan, Wanghan Xu, Wenlong Zhang, Shaofeng Zhang, Yue Zhou, Yifan Yang, Zhihang Zhong, Xue Yang

arXiv:2607.08758v1cs.AI

TL;DR

Existing evaluations rarely test whether AI systems inherit the right mechanisms, repair limitations, and remain coherent with scientific lineages. IG-Bench evaluates lineage reasoning and generation, finding a compositional bottleneck: the best system achieves only 27.3% exact accuracy, while lineage context reshuffles rankings.

  • Problem

    Existing evaluations rarely test whether proposals inherit the correct mechanisms and limitations while remaining coherent with the lineages they extend.

  • Method

    IG-Bench represents papers and proposals with typed Idea Genome objects and GenomeDiff records, then evaluates closed-form reasoning and lineage-grounded generation with PES.

  • Results

    27.3% exact accuracy is achieved by the best system on IG-Exam, while structured lineage context reshuffles system rankings rather than uniformly improving scores.

  • Takeaways & Limitations

    The findings indicate that auto-research systems need compositional verification modules beyond retrieval to produce lineage-coherent scientific ideas.

  • Takeaways & Limitations

    The six evolutionary dynamics are operational evaluation categories rather than an exhaustive theory, and real lineages may mix multiple transition patterns.

Abstract

from arXiv · show

Scientific ideas rarely start from a blank page. They inherit mechanisms, repair known limitations, and recombine pieces of earlier work, much like biological genomes. Current benchmarks still say little about whether AI systems can follow this inheritance structure. We present IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. IG-Bench is organized around the IdeaGene framework: each paper or proposal is represented as a set of minimal, typed, evidence-grounded Idea Genome objects, and a GenomeDiff aligns these objects to record inheritance, mutation, loss, external import, and novel insertion under six operational evolutionary dynamics. The benchmark contains 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records across 10 scientific domains. It supports two evaluations. IG-Exam (42 task types, 1,029 instances) tests closed-form lineage reasoning across Idea Genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification. IG-Arena evaluates generation with a lineage-conditioned Population-Evolution Score(PES), asking whether a proposal can be inserted as a coherent descendant of a given lineage population: it should inherit the right Idea Genome objects, vary meaningfully from nearby work, and offer selection value for future research. Experiments on 14 LLM-based scientists expose a compositional bottleneck. The strongest system reaches only 27.3% exact accuracy on lineage reasoning, and structured lineage context reshuffles system rankings rather than helping every participant uniformly.

1 Introduction

The paper introduces scientific lineage competence and operationalizes it through IdeaGene, a representation and comparison framework for auditing how ideas inherit, mutate, or diverge. It instantiates this framework in IG-Bench, whose reasoning and generation evaluations reveal a compositional bottleneck in current LLM-based scientists.

  • Motivation: Scientific lineage competence requires abstracting papers into Idea Genome objects, tracing their fates, explaining transitions, verifying lineage coherence, and generating lineage-grounded proposals.The target capability distinguishes scientific inheritance from topical or textual similarity.
  • Framework: IdeaGene represents papers or proposals as minimal, typed, evidence-grounded Idea Genome objects and aligns them with GenomeDiff.GenomeDiff records inheritance, mutation, loss, external import, and novel insertion across six operational evolutionary dynamics.
  • Benchmark: IG-Bench contains 1,961 golden lineage traces across 10 scientific domains, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records.The benchmark includes IG-Exam with 42 task types and 1,029 instances, plus IG-Arena for lineage-grounded idea generation under controlled information settings.
  • Empirical findings: 27.3% exact accuracy is achieved by the best system on IG-Exam, exposing a compositional bottleneck in lineage reasoning.Models often recover local signals but fail to keep parent choice, driver assignment, object fate, and verification flags jointly consistent.
  • Empirical findings: Structured lineage context reshuffles system rankings rather than uniformly improving participants, while generated ideas can sound plausible before becoming lineage-coherent.Experiments cover 14 LLM-based scientists, including direct LLMs, research-agent frameworks, and CLI harnesses.

2 Related Work

IG-Bench connects scientific retrieval, reasoning, ideation, citation analysis, and LLM evaluation by making Idea Genome representation and GenomeDiff alignment the central objects. It addresses gaps left by paper-level relevance, unconstrained generation, document-level citation structure, and non-lineage agent benchmarks.

  • Benchmark positioning: IG-Bench combines a representation layer with benchmarking because paper-level relevance is too coarse and generation-only preference is too unconstrained.Its evaluative object is Idea Genome representation and GenomeDiff alignment.
  • Scientific retrieval and QA: Scientific retrieval, embedding, summarization, QA, and autonomous-research benchmarks test relevance, representation, factuality, workflows, or problem-solving, but not the full lineage structure.IG-Bench is positioned between these evaluation traditions by adding lineage-focused representation and reasoning.
  • Automated research and ideation: Automated research and ideation systems increase the need for lineage-grounded evaluation because fluent, literature-aware proposals can still fail to inherit appropriately.Related work spans literature search, hypothesis generation, experimentation, drafting, novelty-aware generation, iterative ideation, retrieval-augmented synthesis, and autonomous collaboration.
  • Science of science and citation structure: Citation and science-of-science research maps fields, communities, paradigm shifts, and citation intent, while IG-Bench aligns method-level entities through GenomeDiff relations.Citation and time serve as candidate evidence, but the evaluative object is the GenomeDiff alignment itself.
  • LLM evaluation methodology: IG-Arena builds on scalable judge and arena protocols while controlling position and length preferences and evaluating scientific lineage rather than general tool use.LLM-as-judge, ELO, and holistic multi-dimensional scoring motivate the methodology, whereas SWE-bench, GAIA, and τ-bench do not test scientific lineage competence.

3 IdeaGene Framework

The IdeaGene framework operationalizes scientific lineage as auditable, mechanism-level inheritance rather than topical similarity. It represents papers as typed Idea Genome objects and uses GenomeDiff alignments plus fixed evolutionary categories to evaluate lineage reasoning and generation.

  • Scientific lineage competence: Scientific lineage competence requires abstracting heritable mechanisms, tracing their persistence or change, explaining transitions, rejecting topical links, and generating plausible descendants.Lineage claims depend on mechanism-level inheritance rather than mere co-location in a shared research setting.
  • Idea Genome objects: An Idea Genome is a minimal, typed, evidence-grounded, lineage-relevant object representing a paper or proposal’s inheritable structure.Objects may encode niches, mechanisms, observations, limitations, deltas, or claims, with content, evidence pointers, and optional constraints.
  • Genome extraction: Genome extraction maps a paper or proposal to auditable objects for later GenomeDiff alignment and lineage verification, rather than producing a general summary.Valid objects must be typed and evidence-grounded, with support from source text or structure such as sections, spans, figures, tables, or equations.
  • GenomeDiff alignment: GenomeDiff aligns predecessor and successor objects by type and semantic role, labeling source objects Inherited, Mutated, or Lost and target objects Novel or External.Each record stores a transition driver, task or domain relation, and evidence-backed rationale, based on LLM-assisted extraction followed by expert audit.
  • Evolutionary dynamics: Evolutionary dynamics are operational categories governed first by driver inheritance and, in ambiguous cases, by the priority Hybridization before Speciation before Niche Competition, with Adaptive Radiation before Mutation when setting shift drives the transition.Without driver inheritance, a relation is co-located rather than lineage; the categories support consistent evaluation without claiming to exhaust scientific development.

4 IG-Bench: Dataset and Evaluation Artifact

IG-Bench is a reusable benchmark built from audited Idea Genome and GenomeDiff lineage data, with complementary evaluations for closed-form lineage understanding and lineage-grounded idea generation. IG-Exam uses exact-match reasoning tasks, while IG-Arena scores proposals as coherent descendants within a lineage population.

  • Benchmark design: IG-Bench has two complementary components: IG-Exam tests closed-form lineage understanding, while IG-Arena tests lineage-grounded idea generation.The shared substrate consists of audited Idea Genome objects and GenomeDiff records, supporting evaluation of both understanding and generation.
  • Dataset construction: IG-Bench contains 1,961 golden lineage traces across 10 scientific domains, covering 1,085 curated Idea Genome objects and 920 pairwise GenomeDiff records.Experts seed landmark and frontier papers, expand 3–7-paper traces through citation links, semantic search, and curation, then extract, align, audit, and programmatically verify the records.
  • IG-Exam: IG-Exam comprises 42 main-challenge task types and 1,029 instances spanning single-genome reading, inheritance tracing, transition explanation, and lineage-claim verification.Exact-match scoring requires every required field to be correct simultaneously, including parent, driver, and object fate.
  • IG-Arena: IG-Arena separates Question, Library, and Lineage settings to distinguish parametric ideation, paper-centric context use, and lineage-structured context use.Lineage settings provide an ordered trace with Idea Genome objects and GenomeDiff evidence.

5 Experiments

Experiments show that lineage reasoning is compositionally difficult, while structured lineage context improves generation unevenly by strengthening heredity rather than variation or selection. Retrieval-oriented scaffolds help inheritance tracing more than verification, and PES better captures lineage-grounded insertion than pairwise preference alone.

  • IG-Exam: 27.3% exact accuracy is the best IG-Exam result, reached by GPT-5.5 + Claude Code, while GPT-5.5 reaches 23.1%.Exact scoring requires all fields to be correct simultaneously, exposing errors that combine correct parent identification with incorrect dynamics or fate labels.
  • IG-Exam: 17.4% is the best T4 Lineage Verification score, below T1 Genome Abstraction at 34.4%, T2 Inheritance Tracing at 37.9%, and T3 Evolutionary Reasoning at 25.3%.T4 requires parent identity, Idea Genome compatibility, driver consistency, and evidence validity, making it closest to reliable descendant generation.
  • System analysis: CLI harnesses improve T2 retrieval and comparison but show little benefit on T3 and nearly vanish on T4, while research agents add little lineage-reasoning capability.The results separate information gathering from compositional consistency checking and show that plausibility can exceed preservation of parent mechanisms or limitation-delta relations.
  • IG-Arena: +4.4 is the median PES gain from Question to Lineage, but GPT-5.5 gains only +2.3 while Kimi-K2-Thinking gains +6.9.The uneven gains diagnose differences in whether systems can use Idea Genome structure rather than uniformly improving every participant.
  • IG-Arena: 84.2 Heredity and 83.6 PES are achieved by Lineage-setting Hybridization, compared with 61.9 Heredity and 69.2 PES for Question-only Mutation.Variation and Selection remain nearly constant, so the PES gap is driven by whether recombination is anchored to explicit parent Idea Genome objects and GenomeDiff evidence.
  • IG-Arena: 0.82 is the Spearman ρ between PES and ELO rankings, with divergence when fluent but lineage-incoherent proposals defeat better-grounded alternatives.PES is therefore used as the primary metric because it measures coherent population insertion, whereas ELO captures pairwise preference and surface appeal.

6 Conclusion

IG-Bench evaluates whether AI systems can identify, verify, and extend inherited scientific mechanisms rather than merely generate plausible research text. Results reveal weak lineage reasoning and a plausibility–coherence gap in generated proposals.

  • 6 Conclusion: IG-Bench uses typed Idea Genome objects, GenomeDiff records, and evolutionary dynamics to make scientific lineage evaluation precise and auditable.The benchmark asks whether proposals form coherent descendants of prior work by preserving inherited mechanisms.
  • 6 Conclusion: 27.3% exact accuracy is achieved by the best system on IG-Exam, while tool scaffolds help T2 Inheritance Tracing but barely improve T4 Lineage Verification.Structured lineage context separates systems rather than lifting them uniformly.
  • 6 Conclusion: PES decomposition exposes a plausibility–coherence gap: proposals may sound novel while failing to preserve the parent mechanism or limitation-delta relation required for coherent descent.This gap shows why plausible research text alone is insufficient for lineage-grounded evaluation.

7 Limitations · Appendix

IG-Bench treats its six evolutionary dynamics as operational categories for auditable evaluation rather than an exhaustive theory of scientific development. Because real successors may combine multiple transition patterns, annotations assign primary drivers for consistency.

  • 7 Limitations: IG-Bench’s six evolutionary dynamics are operational categories, not an exhaustive theory of scientific development.They are designed for auditable evaluation.
  • 7 Limitations: Real scientific lineages can combine multiple transition patterns within a single successor.This limits the completeness of representing development through one primary dynamic.
  • 7 Limitations: IG-Bench assigns primary drivers when successors mix transition patterns.The assignment simplifies complex lineage changes into a dominant category.
  • 7 Limitations: Primary-driver assignments keep annotation consistent across complex successor lineages.Consistency is prioritized over representing every transition pattern equally.
  • 7 Limitations: Primary-driver assignments also keep evaluation consistent despite mixed evolutionary transitions.The operational scheme supports comparable auditing across cases.
  • 7 Limitations: The benchmark’s evolutionary categories should therefore be interpreted as an evaluation convention rather than a complete developmental model.This follows from their operational purpose and the possibility of mixed transitions.

A Main-Challenge Task Inventory · B IG-Bench Examples · B.1 IG-Exam

The main benchmark inventory covers IG-Exam’s four-dimensional challenge set and IG-Arena’s domain-balanced tasks, while worked IG-Exam examples demonstrate abstraction, lineage grouping, evolutionary classification, and structural bridge recovery.

  • A Main-Challenge Task Inventory: 42 task types across 4 capability dimensions comprise IG-Exam’s main challenge, totaling 1,029 instances.Calibration and local-matching diagnostics remain in the code release but are excluded from the paper leaderboard.
  • A Main-Challenge Task Inventory: 30 IG-Arena tasks are balanced across 10 scientific domains in the main run.
  • B IG-Bench Examples: The worked examples span four capability dimensions, presenting full questions, choices, and answers for representative T1–T4 tasks.
  • B.1 IG-Exam: T1 Genome Abstraction asks models to classify an anonymized Idea Genome by contribution type, such as method, dataset, analysis, system, or theory.The representative answer classifies a probabilistic model for antibody indel statistics as a method.
  • B.1 IG-Exam: T2 Inheritance Tracing requires partitioning eight shuffled Idea Genome objects into two ordered lineage groups of four.The example separates a NAS-RL lineage from an Inception lineage based on shared architectural-search and efficient multi-scale-extraction mechanisms.
  • B.1 IG-Exam: T3 Evolutionary Dynamics identifies whether a successor’s relationship to a predecessor reflects mutation, adaptive radiation, hybridization, speciation, or niche competition.The adversarial-example case is labeled mutation because it refines the first-order gradient-sign mechanism while narrowing the niche.
  • B.1 IG-Exam: T4 Structural Integrity tests missing-link recovery by selecting a bridge paper connecting two endpoints in a research lineage.The locomotion example selects Y because its periodic gait framework links model-predictive stability control to later sim-to-real reinforcement learning.

B.2 IG-Arena

IG-Arena evaluates whether generated ideas are coherent descendants of a research lineage by judging inheritance, meaningful variation, and selection value. Its biology example illustrates this criterion: Idea A wins because it more directly extends prior design methods toward experimentally grounded functional validation.

  • Evaluation criteria: IG-Arena asks judges to compare anonymously generated ideas on lineage grounding, meaningful variation, and selection value for future research.The task presents a frontier question and requires choosing A, B, or Tie.
  • Example: Fusion plasma control: The fusion-control example targets cross-device transfer, uncertainty-aware safety, and simulator-to-shot adaptation for ITER-scale deployment.The frontier question combines differentiable plasma simulators, multi-tokamak logs, and constrained RL for robust policies across machines.
  • Example: Protein design: The protein-design example seeks a generative framework that closes the gap between computational binder affinity and selective, stable, functional activity in living cells.The stated frontier question emphasizes in vivo biological function rather than in silico affinity alone.
  • Judge result: Idea A wins because it preserves structure-first generative design while adding assay-grounded active learning and causal components targeting the in silico-to-function gap.The judge also cites Idea A’s more feasible validation plan, whereas Idea B relies on weakly transferable in vivo surrogates and expensive mouse-model validation.

C Evaluation Details … F.1 IG-Exam Main-Leaderboard Heatmap

The additional analyses make evaluation auditable, validate benchmark difficulty, and show that lineage-grounded generation’s main weakness is coherent heredity rather than plausibility or novelty. They also reveal heterogeneous information-setting gains and systematic verification difficulty in IG-Exam.

  • C Evaluation Details: IG-Exam logs record decoding budgets, token counts, response diagnostics, and error classes, separating API failures, formatting omissions, and budget exhaustion from ordinary content errors.The exact-match score remains unchanged by token-budget metadata, which supports auditability across providers and reruns.
  • D Human Agreement: 50 graduate annotators validated GenomeDiff labels, IG-Exam difficulty, and IG-Arena preferences, with human judges reaching 80% agreement with the strongest model-judge panel.Disagreements were adjudicated, ambiguous items excluded, and stratified solving was used to distinguish compositional challenge from label noise.
  • E.1 Domain-Level PES: Domain-level Lineage-setting PES averages across three frontier tasks per domain, exposing domain-specific weaknesses that a single overall score can hide.Strong systems remain relatively uniform across 10 domains, whereas weaker systems show sharper domain-specific failures.
  • E.2 PES Decomposition and Information-Setting Breakdown: Across all 14 participants, Variation consistently exceeds Heredity, confirming a systematic plausibility–coherence gap in lineage-grounded generation.The decomposition separates Heredity, Variation, and Selection, while the per-system view sorts systems by aggregate Lineage PES.
  • E.2 PES Decomposition and Information-Setting Breakdown: Question→Lineage gains are heterogeneous: GPT-5.5 rises from 85.2 to 87.5, while weaker systems benefit more from explicit lineage context.The information-setting breakdown compares Question, Library, and Lineage prompts, and the component view identifies the source of PES differences.
  • E.3 Generated Dynamics Distribution: Question-only prompting yields a broader mix of evolutionary moves, whereas Library and Lineage prompting concentrate generated proposals on Hybridization.Frequency alone cannot establish whether recombination is genetically coherent, motivating the quality-map analysis.
  • E Additional IG-Arena Analysis: The generated-dynamics frequency view is insufficient because the same dynamics label can have different Heredity and PES depending on lineage evidence.This limitation explains why the main text replaces raw dynamics frequencies with a quality map.
  • F.1 IG-Exam Main-Leaderboard Heatmap: IG-Exam’s detailed heatmap reports exact accuracy across T1–T4, with verification remaining the hardest axis.The heatmap provides the detailed closed-form diagnostic view behind the main text’s compact radar profile.

F.2 Error Analysis: Failure Flow

IG-Exam failures are concentrated in evolutionary reasoning and verification, propagating through dynamics and driver fields into exact-match errors. T3/T4 errors dominate, with compositional field interactions driving the largest error classes.

  • Failure localization: Most failures originate in evolutionary reasoning and verification, then propagate through driver and dynamics fields before becoming exact-match errors.The failure-flow analysis localizes the primary bottleneck to higher-level reasoning and verification capabilities.
  • Failure localization: T3 and T4 errors dominate, with the thickest flows passing through dynamics and driver fields.These flows fan out primarily to E1 dynamics misclassification and E2 driver misidentification.
  • Failure localization: Genome-fate and relation errors are the next largest classes, confirming that compositional field interactions—not isolated label failures—drive accuracy bottlenecks.The Sankey structure links higher-tier capability failures to interacting field-family errors and downstream error classes.
Loading 2607.08758v1…