PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

Lishan Yu, Derek Jiu, Qizhen Lan, Xiaoqian Jiang

arXiv:2609.29016v1cs.NEcs.LG

TL;DR

LLM-driven discovery can generate and evaluate candidate designs, but repeated cycles do not ensure cumulative progress or guide which directions to pursue, especially when evaluation is costly. EvoTreeNAD grows a persistent genealogy of architectures and uses descendant evidence to select lineages for further design. It outperforms listed baselines across CIFAR-10/100 and all six MedMNIST-v2 tasks, with controlled CIFAR-10 comparisons favoring its routing approach.

  • Problem

    LLM-driven iterative generation and evaluation do not by themselves ensure cumulative progress or identify which directions to pursue, while costly evaluation constrains exploration.

    Source: §1 · p. 1

  • Method

    EvoTreeNAD grows a persistent genealogy of complete trainable architectures and uses top-percentile descendant-family values to select a lineage whose history informs the next design and implementation.

    Source: §1 · p. 2

  • Results

    EvoTreeNAD outperforms compared NAS and NAD baselines on CIFAR-10/100 and the strongest listed baseline on all six MedMNIST-v2 tasks, with 2.05±0.06% test error on CIFAR-10.

    Source: §1 · p. 2, §5.2 · p. 7, §5.5 · p. 9

  • Takeaways & Limitations

    Under specified lineage-conditioned variation assumptions, sustained top-percentile family values characterize the probability of generating high-reward architectures in stationary regimes.

    Source: §1 · p. 2, §4 · p. 6

    Scaling EvoTreeNAD depends on obtaining informative evaluation feedback at an affordable computational cost.

    Source: §6 · p. 10

Abstract

from arXiv · show

AI-driven scientific discovery accelerates research by autonomously developing solutions and designs. Large language model (LLM) agents support this process through iterative generation and evaluation. Yet these iterations alone do not ensure cumulative progress or establish which directions to pursue next. Costly evaluation further constrains the scope of exploration. Neural architecture discovery brings these challenges together, coupling open-ended design with resource-intensive experimentation. We introduce EvoTreeNAD, a genealogy-guided evolutionary algorithm that constructs trainable architectures without a supplied seed or a hand-specified search space. Starting from an empty root, it grows a persistent genealogy in which each new node represents a complete architecture. Top-percentile values computed from each node and its descendants guide lineage selection. Using the selected design history, an Idea Agent proposes a variant and a Code Agent implements it. Each evaluated variant becomes a child node, expanding the genealogy while providing evidence for subsequent lineage selection. Our theoretical analysis establishes the existence of stationary variation regimes as the genealogy grows. Under specified variation assumptions, sustained top-percentile family values quantify the probability of generating high-reward architectures in these regimes. EvoTreeNAD discovers architectures that outperform the compared NAS and NAD baselines, achieving CIFAR-10/100 test errors of $2.05{\pm}0.06\%$ and $15.09{\pm}0.22\%$. On all six MedMNIST-v2 tasks, the discovered architectures surpass the strongest listed baselines. A controlled CIFAR-10 study further shows that EvoTreeNAD outperforms direct generation, best-of-$N$ greedy continuation, and full-family-mean routing.

1 INTRODUCTION

LLM-driven discovery can generate and evaluate candidate designs, but repeated cycles do not guarantee cumulative progress or identify which directions merit continued work. EvoTreeNAD addresses this by retaining design lineages and using accumulated evidence to guide evolution, with theoretical and benchmark results supporting the approach.

1 INTRODUCTION

  • LLM agents can repeatedly generate, implement, and evaluate candidate solutions, but those cycles alone do not ensure cumulative progress or reveal which research directions to pursue.Costly evaluation constrains both the breadth of exploration and the depth devoted to promising directions.Source: §1 · p. 1
  • Existing iterative refinement, candidate selection, and evolutionary search use evaluated solutions, yet candidate quality alone does not reveal a design's potential for further development.Strong designs may yield weak variants, while initially weaker designs may become competitive through modification.Source: §1 · p. 1
  • EvoTreeNAD preserves complete trainable architectures and their branching histories, using top-percentile descendant-family values to select a lineage that informs the next design and its implementation.Each evaluated variant extends the genealogy and supplies evidence for subsequent lineage selection.Source: §1 · p. 2
  • EvoTreeNAD outperforms the compared NAS and NAD baselines on CIFAR-10/100 and the strongest listed baseline on all six MedMNIST-v2 tasks, with controlled CIFAR-10 comparisons favoring its genealogy-guided approach.The controlled comparisons include direct generation, best-of-N greedy continuation, and full-family-mean routing.Source: §1 · p. 2
  • Under specified lineage-conditioned variation assumptions, sustained top-percentile family values exactly characterize the probability of generating high-reward architectures in stationary empirical variation regimes.The analysis also quantifies how weak outcomes affect full-family means and establishes pathwise equivalence between maximum-family routing and best-of-N greedy continuation.Source: §1 · p. 2

2 RELATED WORK

Prior LLM-based discovery systems combine generation with evaluation and organize attempts through candidate retention, search policies, or evolutionary selection. EvoTreeNAD applies lineage selection to neural architecture discovery while retaining evaluated design histories and descendant evidence.

2 RELATED WORK

3 EVOTREENAD

EvoTreeNAD constructs complete trainable architectures within a persistent genealogy, then uses descendant-family evidence to select lineages for continued evolution. Evaluation, lineage growth, and final candidate selection follow fixed task-specific protocols.

3.1 TASK SETTING

  • A task interface fixes inputs and outputs, targets, data splits, evaluator, and reporting protocol while leaving internal modules, connectivity, and forward computation to evolution.Each architecture is a complete trainable model specification, including its model-defined loss; task data, evaluator, and stage-specific training protocols remain fixed.Source: §3.1 · p. 3

3.2 ARCHITECTURE GENEALOGY AND FAMILY VALUES

  • The genealogy begins with an empty routing root; each non-root node represents an architecture, and each edge records a parent-to-child design change.Node rewards measure architecture performance under a fixed short-budget training and validation protocol, while family values aggregate rewards over descendant families.Source: §3.2 · p. 3
  • EvoTreeNAD uses a top-percentile family value to emphasize high-reward family members and reduce sensitivity to weak variants.The descendant family includes the node itself and every architecture evolved below it.Source: §3.2 · p. 4

3.3 EVOLUTIONARY ITERATION

  • Each iteration routes from the root, follows the highest-valued child when a node is saturated, and stops at the first node with remaining capacity.The selected root-to-node path supplies the design and evaluation history for subsequent variation; later descendant outcomes can redirect routing toward a retained branch.Source: §3.3 · p. 4
  • The Idea Agent proposes designs or modifications from the selected lineage's task requirements and history, and the Code Agent implements a proposal as a complete architecture.Proposals can refine existing structures, explore alternatives, or simplify low-contribution components.Source: §3.3 · pp. 4–5
  • Generated implementations undergo executability and interface checks before short-budget training and validation, with poor candidates eligible for early stopping and bounded retries for invalid scores.After evaluation, each valid architecture and its reward are added as a child, and weaker variants remain available for further evolution.Source: §3.4 · p. 5

3.4 FINAL ARCHITECTURE SELECTION

  • Final selection applies the routing rule to the completed genealogy, then fixes the three highest-discovery-reward architectures on the principal lineage as the full-fidelity candidate set.Candidates are trained from scratch and evaluated on the official test split only after the set is fixed.Source: §3.4 · p. 5

4 DYNAMICS OF GENEALOGY-GUIDED EVOLUTION

EvoTreeNAD’s adaptive genealogy admits stationary variation regimes, and its routing rules relate family values to the production of high-reward architectures. Under specified variation assumptions, top-percentile routing tracks high-reward production, while mean and maximum routing have distinct trade-offs.

Stationary selection-variation

  • Every infinite run admits reachable stationary regimes for the sequence of lineage-conditioned reward laws and realized rewards.These regimes are subsequential limits of empirical occupation measures over time shifts.Source: §4 · p. 6

Family values and recurrent development

  • Under specified lineage-conditioned variation assumptions, each recurrent family’s limiting top-percentile value connects to its high-reward production rate.For top-percentile routing, that rate equals the run’s high-reward frequency in the same reachable stationary regime.Source: §4 · p. 6
  • Recurrent child families can have equal limiting full-family means but different high-reward frequencies, with their gap proportional to the weak-outcome-frequency gap.A family with more weak variants can produce high-reward architectures more often yet receive no higher limiting mean.Source: §4 · p. 6
  • Maximum-family routing is pathwise equivalent to best-of-N greedy continuation: only the highest-immediate-reward child is developed further.Unchosen siblings receive no further development.Source: §4 · p. 6
  • Maintaining a high top-percentile family value as a family grows requires a nonvanishing fraction of high-reward descendants, not just a few early successes.Unlike maximum-family routing, EvoTreeNAD can develop successful designs without making earlier branch choices permanent, so later outcomes can redirect evolution to retained alternatives.Source: §4 · pp. 6–7

5 EXPERIMENTS

Experiments assess EvoTreeNAD on CIFAR-10/100 and six MedMNIST-v2 tasks, alongside controlled CIFAR-10 strategy comparisons and lineage analyses.

5.1 PROTOCOL

  • The evaluation covers CIFAR-10/100 and six MedMNIST-v2 tasks, with controlled comparisons of genealogy-guided evolution on CIFAR-10.Main discovery uses three independent runs per task; budgets are up to 300 iterations for CIFAR-10/100, 100 for CIFAR-10 strategy and agent comparisons, and 80 for MedMNIST-v2.Source: §5 · p. 7, §5.1 · p. 7

5.2 ARCHITECTURE DISCOVERY ON CIFAR-10/100

  • 2.05±0.06% CIFAR-10 test error and 15.09±0.22% CIFAR-100 test error outperform the listed NAS and NAD methods.Table 1 reports the best-performing discovered architecture and a smaller competitive architecture for each dataset.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 COMPARISON OF DISCOVERY STRATEGIES

  • EvoTreeNAD consistently outperforms direct generation, best-of-N greedy continuation, and full-family-mean routing on CIFAR-10.The advantage over full-family-mean routing supports top-percentile selection over averaging family rewards; the comparison uses three independent 100-iteration runs per strategy.Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4 ARCHITECTURE EVOLUTION AND AGENT CONFIGURATIONS

  • Later lineage architectures often outperform initial designs despite intermediate declines, and 300-iteration runs show further gains in validation and full-fidelity performance.Short-budget validation scores correlate positively with full-fidelity performance for the evaluated CIFAR-10 architectures.Source: §5.4 · p. 8, Figure 2 · p. 9
  • Across five Idea and Code Agent configurations, EvoTreeNAD discovers competitive architectures; the best-performing ones use GPT-4.1 for proposals and OSS20B for code realization.Code realization accounts for most agent token usage in these runs.Source: §5.4 · pp. 8–9

5.5 ARCHITECTURE DISCOVERY ACROSS MEDMNIST-V2 TASKS

  • EvoTreeNAD outperforms the strongest listed baseline on all six MedMNIST-v2 tasks, spanning diverse modalities and 2D and 3D classification.The experiments start from an empty root on each task.Source: §5.5 · p. 9, Table 3 · p. 9, Table 4 · p. 9

6 DISCUSSION AND CONCLUSION

EvoTreeNAD uses a persistent architecture genealogy and top-percentile family values to guide evolution, with accumulated descendant evidence allowing successful designs to be developed and alternatives reconsidered. The appendix formalizes this growth process, while the paper notes that scaling depends on affordable, informative evaluation feedback.

6 DISCUSSION AND CONCLUSION

  • EvoTreeNAD’s persistent genealogy uses top-percentile family values to guide lineage selection and accumulated descendant evidence to develop successful designs and reconsider alternatives.The paper connects this process to stationary variation regimes and high-reward production.Source: §6 · p. 10
  • Scaling EvoTreeNAD depends on obtaining informative evaluation feedback at an affordable computational cost.The paper presents accumulated experimental evidence as a way to reconsider the direction of LLM-driven discovery.Source: §6 · p. 10

A DYNAMICS OF GENEALOGY-GUIDED EVOLUTION

  • The appendix provides full theoretical statements and proofs about genealogy growth, family-value rules, stationary variation regimes, and high-reward production.It also compares mean and maximum routing.Source: A DYNAMICS OF GENEALOGY-GUIDED EVOLUTION · p. 13

A.1 FORMAL SETUP

  • The formal process starts with an empty root; each non-root node is an architecture with a bounded reward and fixed child capacity.Routing begins at the root and stops at the first node with remaining capacity; at saturated nodes, family values select the next child, with ties resolved by a fixed order.Source: §A.1 · pp. 13–14
  • The process kernel determines the next architecture and reward, while the Grow update adds that architecture under the selected endpoint and updates retained variation information.Routing policies share context construction, variation and update kernels, capacity rules, and tie rules, but their lineage choices shape later contexts and genealogies.Source: §A.1 · p. 14

A.2 PROPERTIES OF FAMILY-VALUE RULES

The appendix characterizes how family-value rules route evolution and respond to growing descendant families. Maximum-family routing matches best-of-N greedy continuation, while top-k averages limit the influence of individual rewards and require recurring high outcomes to sustain a gap.

A.2 PROPERTIES OF FAMILY-VALUE RULES

  • EvoTreeNAD with maximum-family routing is pathwise equivalent to matched best-of-N greedy continuation, generating the same architectures and rewards under the same initial state and random draws.The greedy process generates a node’s children and continues from the highest-immediate-reward child without returning to unchosen siblings.Source: §A.2 · p. 14
  • For a family of n rewards, the top-k family value is the average of its k highest rewards, where k = max(1, ceil(τn)).The family value changes by at most |δ|/k when one reward changes by δ.Source: §A.2 · pp. 14–15
  • Maintaining a family value a fixed amount above q as the family grows requires a nonvanishing fraction of rewards at least q.Finitely many early high rewards cannot preserve that gap if subsequent rewards are at most q.Source: §A.2 · p. 15
  • The support bound relates a top-k family value above q to the number of family rewards at least q.When fewer than k rewards reach q, the top k include all such rewards and the remaining values are bounded by q.Source: §A.2 · p. 15

A.3 STATIONARY SELECTION-VARIATION

The appendix analyzes the long-run variation and routing behavior of genealogy growth. It establishes stationary empirical regimes and, under stated variation assumptions, relates top-percentile family values to high-reward production while distinguishing top-percentile, mean, and maximum routing.

A.3 STATIONARY SELECTION-VARIATION

  • In a stationary regime, the generated reward distribution is the mixture of lineage-conditioned reward laws, and average top-percentile means give a lower bound on high-reward probability.If those top-percentile means average above q, the theorem gives a positive lower bound on the probability of rewards at least q.Source: §A.3 · pp. 15, 17
  • Every infinite genealogy-growth realization admits weak subsequential limits of its empirical occupation measures, and every such limit is shift-stationary.A reachable stationary regime tracks the conditional reward laws and realized rewards along the run.Source: §A.3 · p. 16

A.4 TOP-PERCENTILE ROUTING AND HIGH-REWARD PRODUCTION

  • Under stated variation-law conditions and a convergence assumption, top-percentile family values of recurrent families characterize high-reward production in stationary regimes.Maintaining a fixed gap in a growing family requires a nonvanishing fraction of high-reward descendants, rather than only isolated early successes.Source: §A.4 · p. 21
  • Theorem 2 distinguishes routing rules: recurrent top-percentile child families share high-reward frequency, while full-family means can assign equal limiting values to families with different high-reward rates.Maximum-family routing is pathwise equivalent to matched best-of-N greedy continuation.Source: §A.4 · p. 21
  • Full-family means depend on weak outcomes, whereas maximum-family routing selects the winning child at each generation.The paper states that accumulated descendant evidence can also redirect evolution toward retained alternatives.Source: §A.4 · p. 21

B.1 HYPERPARAMETERS

  • Agent context for architecture variation includes the selected architecture and evaluation results, plus records from up to kidea recent ancestors for the Idea Agent and kcode for the Code Agent.The limits retain recent design history while controlling context length.Source: §B.1 · p. 21
  • EvoTreeNAD can perturb sibling family values slightly at random before branch selection to encourage exploration of alternative lineages.Source: §B.1 · p. 21

B.2 DISCOVERY-STAGE EVALUATION

The discovery-stage evaluation screens generated architectures and uses short-budget validation with early stopping to limit evaluation effort. The appendices describe the required model interface and report the discovery protocols and agent configurations used in experiments.

B.2 DISCOVERY-STAGE EVALUATION

  • Generated implementations must pass execution and task-interface checks before short-budget evaluation; failed attempts receive diagnostic feedback, with bounded retries and parent-specific tolerance for failed expansions.Expansions that continue to fail are recorded under the parent with a fixed negative reward after the tolerance is exceeded.Source: §B.2 · pp. 21–22
  • Discovery uses fixed short-budget training on the task validation set, with early stopping when a candidate falls below a threshold based on the strongest prior candidates’ scores at the same training step.Recorded validation results, including those before early stopping, provide discovery scores for node rewards.Source: §B.2 · p. 22

C MODEL INTERFACE

  • The CIFAR-10 model interface specifies forward for training and predict for evaluation, with required tensor inputs and outputs; extra unknown keyword arguments must be tolerated.Evaluation uses predict logits and the task metric, while forward returns the training loss.Source: C MODEL INTERFACE · pp. 22–23

D.1 DISCOVERY PROTOCOL

  • Main discovery uses three independent EvoTreeNAD runs per task; MedMNIST-v2 runs use 80 iterations, while CIFAR-10 strategy and agent comparisons use 100 iterations per run.Main results use GPT-4.1 for architectural proposals and OSS20B for code realization.Source: §D.1 · p. 23

D.2 MEDMNIST-V2 TASKS AND DATA

The experiments cover six MedMNIST-v2 tasks, specify a full-fidelity evaluation protocol, and examine additional EvoTreeNAD analyses and representative discovered architectures.

D.2 MEDMNIST-V2 TASKS AND DATA

  • Six MedMNIST-v2 tasks span 2D images and 3D volumes across imaging modalities, including pathology, retinal, kidney-cell, vessel, synapse, and organ classification.The tasks use the official train, validation, and test splits.Source: §D.2 · p. 24, Table 7 · p. 24

D.3 FULL-FIDELITY EVALUATION PROTOCOL

  • Discovered architectures are retrained from scratch on each dataset’s full official training split, with task-specific fixed regularization and augmentation.The full-fidelity settings are summarized in Table 8; all runs use AdamW and a cosine-decay learning-rate schedule.Source: §D.3 · p. 24, Table 8 · p. 24

E ADDITIONAL EXPERIMENTAL ANALYSES

  • Additional analyses examine representative architectures, principal-lineage performance, continued evolution, discovery strategies, and agent configurations and costs.Source: E ADDITIONAL EXPERIMENTAL ANALYSES · p. 24

E.1 REPRESENTATIVE DISCOVERED ARCHITECTURES

  • Table 9 summarizes structures of representative architectures discovered across the evaluated task families, starting from an empty root.Source: §E.1 · p. 24, Table 9 · p. 25

E.2 ARCHITECTURE PERFORMANCE ALONG PRINCIPAL LINEAGES

The lineage analysis evaluates selected architectures at early and later stages under full-fidelity settings, finding variable performance and useful short-budget screening signals.

E.2 ARCHITECTURE PERFORMANCE ALONG PRINCIPAL LINEAGES

  • Three architectures are evaluated per principal lineage: the first after the root, the highest-reward architecture in the first half thereafter, and the highest-reward architecture in the second half.These selections span early and later lineage stages and are assessed using the full-fidelity protocol.Source: §E.2 · p. 25
  • Performance rises and falls across evaluated lineage architectures, with best results frequently later and parameter counts often decreasing alongside modest accuracy gains.Source: Figure 3 · p. 25, §E.2 · pp. 25–26
  • Empty-root architectures can already be competitive, while later architectures more consistently achieve high performance.Source: §E.2 · p. 26
  • Short-budget and full-fidelity performance have a positive relationship for the selected CIFAR-10 architectures, supporting short-budget scores as a useful screening signal at this benchmark scale.Source: §E.2 · p. 26

E.3 CONTINUED ARCHITECTURE EVOLUTION

Continued evolution is examined in two extended CIFAR-10 runs, while the accompanying strategy illustration distinguishes lineage routing from alternative discovery procedures.

E.3 CONTINUED ARCHITECTURE EVOLUTION

  • Later evolution with a GPT-4.1 Idea Agent found the best-performing CIFAR-10 architecture; the run without an Idea Agent improved more gradually, and both trajectories were non-monotone.The two runs were continued to 300 iterations, and full-fidelity evaluations selected architectures whose discovery rewards improved over their immediate predecessors.Source: §E.3 · p. 26

E.4 ILLUSTRATION OF DISCOVERY STRATEGIES

  • EvoTreeNAD retains branches and routes by top-percentile family values, unlike greedy continuation, independent direct generation, or full-family-mean routing.Greedy continuation keeps only the highest-immediate-reward child; direct generation repeatedly starts at the empty root.Source: §E.4 · p. 26

E.5 AGENT CONFIGURATIONS AND COST

The agent study compares proposal and implementation configurations, reports a practical pairing and costs, and provides distinct roles for the Idea and Code Agents.

E.5 AGENT CONFIGURATIONS AND COST

  • The configuration study holds lineage selection fixed across five agent setups, using three independent 100-iteration runs per setup.Table 10 summarizes generation statistics, discovery-stage performance, resource use, and cost; attempts include code-realization retries.Source: §E.5 · p. 26, Table 10 · p. 28
  • The highest full-fidelity accuracy in the compared runs came from GPT-4.1 proposals paired with OSS20B code realization.The authors identify this as a practical pairing of a stronger proposal model with a lighter local implementation model.Source: Figure 5 · p. 27, §E.5 · pp. 27–28
  • A 100-iteration CIFAR-10 run uses approximately 0.27–0.37 GPU-days, while GPT-4.1 proposals cost less than $1 and GPT-5 code realization approximately $3 in API fees.Candidate evaluation is capped at 10.5 minutes; adaptive early stopping reduces evaluation time, and local OSS20B incurs no API fees.Source: §E.5 · p. 28

F.1 SYSTEM PROMPT FOR IDEA AGENT

  • The Idea Agent reviews prior designs and metrics, then proposes multiple distinct upgrades, including exploratory or subtractive changes.Source: §F.1 · pp. 28–29

F.2 SYSTEM PROMPT FOR CODE AGENT

  • The Code Agent implements the selected proposal, audits prior designs and metrics for improvement opportunities, and outputs upgraded model code.Source: §F.2 · pp. 30–31

Related papers

Loading 2609.29016v1…