PaperlayerView exact revision on arXiv

Source-linked AI summary

Evolutionary Ensemble Search: Council-Guided Program Evolution with Persistent Memory

Juan P. Madrigal-Cianci, Eshan Chordia

arXiv:2609.17590v1cs.NEcs.AI

TL;DR

Machine-learning engineering requires choices about data, representations, procedures, and predictions, and search must preserve alternatives and learn from observed errors. EES joins expert-guided directions with executable evolutionary changes and persistent memory. Its development ledger records medal-threshold artifacts on 19 of 22 MLE-bench Lite tasks, while the paper cautions that this is not evidence that the full architecture produced every artifact.

  • Problem

    Machine-learning engineering requires choices about data, representations, fitting procedures, and prediction construction, while effective search must preserve alternatives and retain useful experience.

    Source: §1 · p. 1

  • Method

    EES guides parent-conditioned program mutation and crossover with structured expert review, evaluates changed procedures, and retains experience through operator credit and memory.

    Source: §1 · pp. 1–2

  • Results

    19 of 22 tasks (86.36%) have medal-threshold artifacts in the MLE-bench Lite ledger, with best outcomes of 11 gold, five silver, and three bronze.

    Source: §5.3 · p. 10

  • Takeaways & Limitations

    EES specifies an approach to cumulative executable search in which measured procedures, their ancestry, and their measurements form the reusable unit.

    Source: §6 · p. 11, §7 · p. 12

    The Lite record does not establish that the full council, nested evolution, and memory stack jointly produced every artifact or estimate their marginal benefit.

    Source: §6 · p. 11

Abstract

from arXiv · show

Evolutionary Ensemble Search (EES) constructs machine-learning procedures through expert-guided program evolution. A role-specialized council turns task evidence and experimental results into structured search directions. An orchestrator allocates these directions to execution specialists and an evolutionary engine. The engine selects measured parents, diagnoses their errors, and produces descendants through code mutation, structured pipeline edits, and crossover. Each child must execute and acquire its own validation evidence. Population archives retain useful alternatives, while compatible predictions compete in a validation-gated ensemble stage. Search adapts through parent-relative operator credit, session memory, and problem-indexed lessons retrieved across runs. We specify these mechanisms, distinguish their execution profiles, and define the contracts required to compare candidates as their computations change. A public MLE-bench Lite development ledger records medal-threshold artifacts on 19 of 22 tasks (86.36\%), with best outcomes of 11 gold, five silver, and three bronze. The procedures span text, images, tables, audio, scientific geometry, and deterministic transformations. The campaign includes grade feedback between runs, external-source routes, and mixed confirmation procedures; its aggregate is an achieved development result, not a blind autonomous-agent success rate. The report contributes a concrete architecture for cumulative executable search and a versioned account of its cross-modal development outcomes.

1 Introduction

Machine-learning engineering search must preserve alternatives, connect observed errors to subsequent edits, and retain useful experience. EES combines expert-guided directions with executable evolution and memory, while its Lite study measures achieved solution breadth and performance rather than component contributions.

1 Introduction

  • Machine-learning engineering choices include data, representation, fitting procedures, and prediction construction; search agents address them through code generation, refinement, and repeated execution.Effective search must also preserve alternatives, connect errors to later edits, and retain experience beyond the current experiment.Source: §1 · p. 1
  • EES combines role-specialized council review, bounded orchestration, and evolution that selects measured parents, evaluates changed procedures, and retains candidates.Council roles include discovery, methodology, memory, modeling, and implementation review; the council structures directions rather than voting on a final answer.Source: §1 · p. 1
  • A proposed mutation has no fitness until its child executes under the comparison protocol, separating language-model-guided proposals from measured outcomes.Observed residual error, a train–validation gap, or a weak component can motivate a testable change.Source: §1 · p. 1
  • EES adapts at three timescales through local operator rewards, session memory, and a persistent task-indexed library, without updating language-model weights.The library associates task fingerprints with lessons, strategy statistics, and adaptable candidates.Source: §1 · pp. 1–2
  • The architecture specifies parent-conditioned mutation, crossover, screening, archive retention, and prediction-level recombination, alongside formal accounts of inheritance, comparability, operator credit, and memory.It distinguishes execution profiles rather than treating all runs as using identical policies.Source: §1 · p. 2
  • The paper presents a versioned MLE-bench Lite campaign analysis covering task scores, confirmation identities, representative procedures, and reproducibility scope.This is one of the stated contributions, alongside the architecture and formal account.Source: §1 · p. 2
  • Algorithmic discovery searches executable procedures and their computational structure, whereas scientific novelty additionally requires comparison with prior methods.The Lite study measures achieved solution breadth and performance, not the contribution of each search component.Source: §1 · p. 2

2 Related Work

EES builds on execution-driven machine-learning agents, evolutionary program search, AutoML, ensembles, and reflective memory. Its organizing distinction is how structured review, population evolution, operator feedback, and cross-run retrieval interact.

Machine-learning engineering agents

  • EES shares execution-driven problem solving with MLE-STAR and MARS, but organizes search around expert review, a persistent population, operator feedback, and cross-run retrieval.MLE-STAR refines influential code blocks and ensembles candidates; MARS combines cost-constrained tree search, modular implementation, and comparative memory.Source: §2 · p. 2

Evolutionary program search

  • EES applies evolutionary program search to complete ML procedures, using typed configuration changes and diagnostic code edits with data, prediction, and validation semantics for comparison.Related approaches include genetic programming, FunSearch, and AlphaEvolve [6].Source: §2 · p. 2

AutoML and predictive ensembles

  • EES incorporates established pipeline search, meta-learning, model reuse, and validation-based ensemble selection, while distinguishing prediction recombination from program crossover.Its procedures can also include generated code, deterministic transformations, scientific feature construction, and relationship-based methods.Source: §2 · p. 2

Reflective memory

  • EES separates reflective textual memory from numerical operator credit and task-indexed cross-run records, which influence different stages of subsequent search.Reflexion uses feedback-derived text without weight updates [10], while MARS derives comparative lessons from search branches [2].Source: §2 · p. 2

3 Search Objects and Measurement Contracts

EES represents tasks, executable procedures, evaluation protocols, and evolving search state separately. Its measurement contracts determine which results are comparable and which candidates remain eligible for search.

3.1 Task and executable candidate

  • A task specifies training and test data, metric, output contract, and resource limits; a candidate records data access, representation, model structure, fitting, inference, and post-processing.Candidate records also retain ancestry, execution status, diagnostics, measurements, and artifact identities.Source: §3.1 · p. 3
  • Candidate scores are comparable only under a common protocol defining evaluation rows, target semantics, metric computation, and information available during fitting.Changing a split alone does not establish improvement; both procedures must be reevaluated on a shared protocol.Source: §3.1 · p. 3

3.2 Execution, eligibility, and cost

  • Execution maps candidates to predictions and result records; only usable, comparable evidence defines a score, while timeouts, exceptions, and invalid artifacts are typed failures.A sample-shaped fallback can meet the output contract without entering the validation-ranked pool.Source: §3.2 · p. 3
  • Search cost includes failed work and resource use beyond selected-candidate training, while deadlines and caps on generations, population, parents, and offspring bound execution.Parallel resource use is distinct from elapsed wall time.Source: §3.2 · p. 3

3.3 Population state and implementation profiles

  • Search state includes an evaluated population, reusable archive, session memory and retrieved context, mutation distribution, and remaining budget; the execution log separately retains failed candidates.The three profiles share candidate, evaluation, and lineage interfaces but use different search policies; the nested profile supplies the detailed adaptive policy.Source: §3.3 · p. 4, Table 1 · p. 4

4 Council-Guided Evolutionary Search

EES couples a role-specialized council and orchestrator to executable evolutionary search: proposed changes become candidates through execution, and measured evidence guides selection and later search.

4.1 Council, orchestration, and initial evidence

  • Council experts provide complementary records, while the orchestrator routes specialist work; recommendations remain advisory and executable evaluations govern selection.Council reviews can redirect multiple subsequent generations.Source: §4.1 · p. 5

4.2 Directed mutation: from parent evidence to child program

  • Mutation uses a measured parent’s diagnostics to propose code or typed-pipeline edits, and every child must execute anew to acquire validation evidence.Unchanged components may be inherited, but inherited program fragments do not carry fitness.Source: §4.2 · p. 5
  • Mutation neighborhoods vary by profile, and fresh drafts, single-parent mutations, and multi-parent crossovers are distinct; admission does not require outperforming the parent.A child can be retained for diversity, quality–complexity tradeoffs, or ensembling; a measured improvement applies only to its shared comparison protocol.Source: §4.2 · p. 6

4.3 Selection, archives, and nested scheduling

  • Nested selection balances quality with lineage diversity, while archives preserve non-dominated quality–complexity alternatives and scheduling separates retrieval, diagnosis, and candidate evaluation.Archived candidates may remain eligible for crossover after leaving the active population; reduced-fold screening does not replace full-evaluation fitness.Source: §4.3 · pp. 6–7

4.4 Operator credit and persistent memory

  • Parent-relative operator credit rewards attempts passing a configured improvement gate, while session and task-indexed memories inform later proposals and retrieved procedures are reevaluated.The paired-fold gate is a search heuristic, not a confirmatory statistical test.Source: §4.4 · pp. 7–8

4.5 Predictive recombination and finalization

  • Predictive recombination requires matching row, target, class, and protocol semantics; an ensemble replaces the best single procedure only through a common-validation gate.Repeated internal validation selection can overfit, so final generalization requires evaluation outside that process.Source: §4.5 · p. 8

4.6 Execution record and terminal grading

  • Structured execution records distinguish intended edits from executed changes and preserve lineage and outcomes; the algorithm screens where configured, executes eligible children, and updates survivors and archives.Reconstruction depends on referenced programs, inputs, dependencies, and model settings being available.Source: §4.6 · pp. 8–9
  • The procedure is frozen before optional external grading; development mode may feed the grade into later engineering runs, whereas evaluation mode treats it as terminal for the frozen run.Official grading is disabled by default and follows terminal artifact selection when requested.Source: §4.6 · pp. 8–9

5 MLE-bench Lite Development Study

The MLE-bench Lite study reports a development ledger across 22 tasks, with recorded outcomes, confirmation records, and representative procedures. Its evidence is bounded by mixed confirmation types, external-source routes, and incomplete campaign-wide provenance.

5.1 Benchmark, evidence unit, and protocol

  • C1–C3 may be new seeds, separate process executions, or exact prediction-file replays, while grades fed later runs and the ledger omits attempted candidates and consolidated compute.The record also lacks artifact-by-artifact activation records for council review, online credit, and cross-run retrieval.Source: §5.1 · p. 9

5.2 External sources and measurement scope

  • Four routes use external sources or historical lookup, and retrieved targets do not establish predictive generalization; numerical grading alone also does not establish protocol compliance.Removing the four explicit flags would not certify the remaining tasks without a source and trajectory audit.Source: §5.2 · pp. 9–10

5.3 Aggregate outcomes and confirmations

  • 19 of 22 tasks (86.36%) have medal-threshold artifacts, with best outcomes of 11 gold, five silver, and three bronze.The three tasks without confirmed medals have null score entries, which do not distinguish below-threshold scores from other terminal outcomes.Source: Table 4 · p. 10, §5.3 · p. 10
  • All three confirmation columns retain medals on the same 19 tasks, but medal levels differ; the 57 medal-bearing records represent 46 task–submission-hash pairs.Repeated hashes and mixed confirmation procedures mean the columns are not uniformly independent repetitions of a complete search.Source: §5.1 · p. 9, §5.3 · p. 10

5.4 Representative executable procedures

  • Representative procedures span geometry-derived ExtraTrees features, pixel-wise image reconstruction, deterministic transformations, and external retrieval, showing computation beyond estimator hyperparameters.The ledger describes recorded solutions rather than reconstructed search trajectories.Source: §5.4 · pp. 10–11, Table 5 · p. 11

6 Discussion and Limitations

The discussion distinguishes EES’s specified architecture from what the Lite campaign establishes. It identifies limits on causal attribution, transfer claims, and scientific-novelty claims, and outlines evaluation priorities.

Discussion and limitations

  • EES’s reusable unit is a procedure together with the problem, ancestry, and measurements needed to assess it.Council review and diagnostic intervention guide where to search, while mutation, crossover, validation, archives, and memory govern procedure changes and reuse.Source: §6 · p. 11
  • The Lite record documents cross-modal solution construction and achieved threshold performance, but does not establish that the full EES stack produced every artifact or estimate its components’ marginal benefits.It also does not estimate a uniform cost advantage or blind generalization to untouched tasks; per-run manifests are needed to link configurations and artifacts.Source: §6 · p. 11
  • Parent-relative rewards are operational credit, not causal attribution, and repeated adaptive selection can exploit validation noise.Mutation selection, reduced-fold screening, archive reuse, and ensemble fitting all act on finite validation data.Source: §6 · p. 11
  • A controlled component study should use shared seeds, fixed protocols, matched budgets, and separate ablations; transfer evaluation requires a pre-library task split without evaluation-task feedback.These are evaluation priorities, not results of the present study.Source: §6 · p. 11
  • Structural change alone does not establish scientific novelty; discovery claims need a new method, problem family, closest prior methods, and mathematical or empirical support.Lineage identifies a procedure but does not prove correctness, convergence, or robustness.Source: §6 · p. 12

7 Conclusion

EES combines expert review, evolutionary changes to measured procedures, predictive ensembling, and persistent memory. The conclusion pairs this architecture with a Lite development record, while noting that the engine itself is proprietary.

Conclusion

  • EES edits measured parents, executes children under comparable protocols, and uses results to guide subsequent search.Operator credit and task-indexed lessons carry feedback within and across runs.Source: §7 · p. 12
  • The MLE-bench Lite development record documents medal-threshold artifacts for 19 of 22 tasks across markedly different procedures, within an implemented cumulative ML-procedure search architecture with generation, evaluation, and reuse interfaces.Source: §7 · p. 12
  • The pinned public repository provides the ledger, solution notes, selected artifacts, and reproduction instructions, but not the proprietary EES engine.Appendix C distinguishes artifact checking from complete search reproduction.Source: §7 · p. 12

A Execution Settings and Benchmark Operators

The method identifies the settings needed for a search rerun, but the campaign ledger does not consolidate them for every task.

Execution settings and benchmark operators

A.1 Benchmark operator catalog

The benchmark operator catalog describes callable entry points and distinguishes them from the broader procedures used in campaign recipes. Its mutation and prediction interfaces record candidate changes and compatibility information, while stated interface properties do not establish convergence or generalization guarantees.

A.1 Benchmark operator catalog

  • The benchmark profile exposes 12 callable entry points, but campaign recipes can contain additional implementations and a single entry point may contain an estimator portfolio.The catalog should not be read as a one-to-one inventory of campaign solutions.Source: §A.1 · p. 13
  • Benchmark variants change configured seeds, text solvers or portfolios, tabular model families, or media training allocation, with runtime records tracking generation, ancestry, changed fields, and rationale.Prediction registration separately records validation kind, row identity, target order, output shape, and artifact locations.Source: §A.1 · p. 13
  • Prediction recombination can average selected predictions or fit weights on compatible validation arrays, but runtime comparability checks do not formally guarantee model correctness or input admissibility.The contracts are runtime checks, not proofs of correctness.Source: §A.1 · p. 13
  • The interface preserves probability-row normalization under Eq. (14)'s hypotheses and a nondecreasing best recorded validation score when the incumbent and protocol remain unchanged.Neither property provides a program-search convergence theorem or a generalization-error bound; protocol changes or score reevaluation break the retention argument.Source: §A.1 · p. 13

B Complete Lite Results

The complete Lite results section reports retained best scores and mixed confirmation records, while emphasizing that these records do not uniformly represent independent repetitions. It also identifies the public evidence packet and limits what its identifiers establish about reproducibility.

B Complete Lite Results

  • C1–C3 are mixed confirmation procedures, not uniformly independent repetitions of a complete search, and the best score can fall outside those columns.A dash indicates a null entry rather than an imputed numerical loss; hash counts identify distinct prediction-file identities, not independent model fits.Source: B Complete Lite Results · p. 14
  • All three confirmation columns retain a medal on the same 19 tasks, but medal levels differ; NOMAD shifts from gold in C1 to silver in C2–C3, while APTOS's best silver is not reproduced at silver.Seven tasks have a best score distinct from every confirmation score at recorded precision; the table distinguishes best artifacts from confirmation performance.Source: Table 7 · p. 14, B Complete Lite Results · p. 14

C.1 Public evidence identity

  • The ledger's public evidence commit identifies the evidence packet, but printing a digest neither establishes its preimage's availability nor identifies the private engine revision.The packet reports board and manifest identifiers; reproducing their exact bytes requires access to the referenced material.Source: §C.1 · p. 15

C.2 Artifact-specific execution provenance

Artifact-specific provenance distinguishes recorded scores and files from claims about how procedures were executed. Reproduction also has separate levels: checking a ledger, regrading an artifact, and reproducing the search require different evidence and resources.

C.2 Artifact-specific execution provenance

  • Different assisted-reproduction and aggregate scores do not establish whether assistance was removed, and earlier assistance labels do not automatically classify later artifacts.Execution claims require the associated program, run configuration, intervention record, and terminal artifact identity.Source: §C.2 · p. 15
  • A replay must match the exact submission and grade record, not merely a directory name or rounded score; artifact availability also differs, and the Dirty Documents prediction file is not committed.Some solution directories store an earlier artifact while their notes quote the aggregate ledger.Source: §C.2 · p. 15

C.3 Three reproduction levels

  • Ledger checking, artifact regrading, and search reproduction are distinct levels with progressively different requirements, including exact prediction files and benchmark data for regrading and implementation, dependencies, compute, and profile state for search reproduction.The grading command presupposes benchmark setup and data access and does not reproduce the search.Source: §C.3 · p. 15
Loading 2609.17590v1…