PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Reflective Evolution Effectively Improves Production AI Systems

Seung Won Wilson Yoo, Chang Zeng, Aditi Somayajula, Sahithi Ankireddy, Rohan Kosalge, Avishree Khare, John Mitchell, Aditya G. Parameswaran, Rajhans Samdani, Anupam Datta

arXiv:2610.12794v1cs.LGcs.AIcs.CLcs.SE

TL;DR

Production AI failures can conceal interacting causes and relevant evidence across an evolutionary search history, limiting one-shot, parent-local mutation. CoCoEvolve combines historical retrieval with live-system probing and mutation validation. Across production tasks, it outperforms evaluated baselines, including on held-out data-eng-bench.

  • Problem

    Most LLM-guided evolutionary optimizers mutate one selected parent from its failures in a single pass, although production-system failures may hide interacting causes and relevant evidence across search history.

    Source: §1 · p. 1

  • Method

    CoCoEvolve retrieves evidence across evolutionary history and probes the live target while its coding-agent mutator constructs and validates mutations.

    Source: §1 · pp. 1–2, §2.2 · p. 4

  • Results

    Across production AI tasks, CoCoEvolve outperforms evaluated baselines, solving 77.4% of held-out data-eng-bench tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results suggest that curated historical evidence, on-demand tools for probing the live environment and search history, and a coding-agent mutator offer an effective path to improving real-world AI systems.

    Source: §5 · p. 10

    At submission, the DABStep score had not been externally validated, and the authors had not received a response from benchmark maintainers.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large Language Model (LLM)-guided evolutionary optimization has successfully improved prompts, programs, and even entire AI systems. Most optimization methods select a candidate from the search frontier, then mutate it in one LLM pass based only on its failures. This focus on the frontier can stall on production AI systems, where isolated failures reveal little about complex, hidden interactions. We introduce CoCoEvolve (Cortex Code Evolve), an evolutionary optimizer built around what we call meta-reflection, wherein an agentic mutator (Cortex Code) selectively retrieves evidence from the entire evolutionary history and probes the live system before verifying and proposing mutations. We apply CoCoEvolve to optimize three classes of production AI systems: data agents, LLM-powered data pipelines, and data engineering workflows. On DABStep, CoCoEvolve tops the public leaderboard, reaching 92.8% held-out hard accuracy compared to 78.4% for Meta-Harness and 30.4% for GEPA under matched conditions. Across six LLM data pipeline workloads, it outperforms every evaluated baseline. On data-eng-bench, CoCoEvolve lifts test-set solve rate from 59.1% to 77.4%, past the best baseline of 73.6%. It also has 11 percentage points higher accuracy than GEPA on HotpotQA while remaining competitive with specialized optimizers on established mathematical and systems benchmarks. Matched ablations confirm that each component of CoCoEvolve contributes to these gains, demonstrating the value of meta-reflection for optimizing production AI systems.

1 INTRODUCTION

LLM-guided evolutionary optimizers often make one-shot, parent-local edits, which can miss causes distributed across search history or hidden in system interactions. CoCoEvolve instead uses historical evidence and live-system probing to optimize production AI systems and is evaluated across data agents, LLM pipelines, data-engineering workflows, and standard benchmarks.

1 INTRODUCTION

  • CoCoEvolve retrieves evidence beyond the selected parent's failures and equips a coding-agent mutator to inspect, experiment, revise, and validate a child before committing it.History access includes both a compact summary and structured queries into prior programs and mutations.Source: §1 · pp. 1–2
  • In the Figure 1 example, combining a prior program trace with documentation raises training accuracy from 28/300 to 95/300.The mutator tests its hypothesis before committing the repair.Source: Figure 1 · p. 2
  • The evaluation covers data agents, LLM-powered data-processing pipelines, data-engineering workflows, and standard prompt, mathematical, and systems benchmarks.The standard benchmarks test competitiveness where environment interaction is limited or unnecessary.Source: §1 · p. 2
  • DABStep held-out hard accuracy is 92.8% for CoCoEvolve, versus 78.4% for Meta-Harness and 30.4% for GEPA; it also improves six pipeline workloads.The pipeline improvement is up to 10 pp over the strongest evaluated baseline.Source: §1 · p. 2
  • At submission, the DABStep leaderboard score had not been externally validated, and the authors had not received a response from benchmark maintainers.The authors had contacted the maintainers.Source: §1 · p. 2
  • CoCoEvolve leads held-out data-eng-bench, solving 77.4% of tasks and achieving the strongest overall performance among evaluated evolutionary methods.Matched ablations confirm that each component contributes to the gains.Source: §1 · p. 3

2 COCOEVOLVE

CoCoEvolve optimizes text-representable artifacts by using accumulated search history and target-environment feedback to guide mutation. Its search policy combines parent selection with optional task selection that directs effort toward informative, potentially transferable failures.

2.1 PROBLEM FORMULATION

  • The optimization target is a text-representable program whose aggregate fitness is expected score over a task distribution, with observations recording evaluation traces such as tool calls and errors.The archive stores evaluated candidates, and each iteration selects a parent and produces a child.Source: §2.1 · p. 3

2.2 META-REFLECTION

  • Meta-reflection retrieves selected evidence from the accumulated search history and acquires feedback from the target environment while constructing and validating mutations.The mutator receives a curated context bundle and can query the environment on demand before committing a child.Source: §2.2 · p. 4
  • A compact ledger tracks recent strategies and outcomes, while live tools support execution, testing, and target-system queries before a child enters outer-loop evaluation.On-demand history queries can retrieve task lineage, capability-restoring diffs, and known co-regressions.Source: §2.2 · p. 4

2 COCOEVOLVE

  • Each iteration returns the validated child to the outer search loop, where it is scored, recorded in history, and considered under the archive-admission and parent-selection policy.Source: §2.2 · p. 4

2.3 SEARCH POLICY

  • The default parent selector is standard MAP-Elites; a novel budgeted ℓp roster is introduced for question or test-case pools to prioritize complementary coverage.An optional similarity-informed task-selection policy uses proxy scores to focus mutation on high-information tasks.Source: §2.3 · p. 4
  • Sparse evaluation uses a semantic task graph to estimate scores on untested tasks, then directs effort toward unresolved targets with broad estimated transfer.Information-gain sampling focuses on likely failures with broad semantic reach and downweights repeatedly unsuccessful targets.Source: §2.3 · p. 5

3 EXPERIMENTS

The experiments compare CoCoEvolve with matched baselines across production agents, LLM-powered pipelines, data engineering, and established mathematical, systems, and prompt-optimization tasks. They report gains across several production settings, alongside competitive results on established benchmarks.

3.1 AGENT OPTIMIZATION

  • 95.24% hard accuracy on DABStep’s official 450-question suite puts CoCoEvolve atop the validated public leaderboard.The agent-optimization comparison also reports that CoCoEvolve outperforms GEPA and Meta-Harness under matched splits and budgets.Source: §3.1 · p. 6, Table 1 · p. 6
  • On DABStep, live SQL tests exposed two hidden fee-logic errors and enabled a verified vectorized fee engine, producing the run’s largest single gain, from 28 to 95 correct answers.A separate matched run found GEPA left the single-row fee calculator unchanged, whereas CoCoEvolve tested structural remedies including a stored procedure and dynamic table.Source: §3.1 · p. 6

3.2 AI FUNCTIONS

  • On Lockpicking, CoCoEvolve reaches exact recovery at iteration 6, while AdaEvolve’s best score remains 90% after its first plateau-triggered paradigm generation.The account attributes the recovery to querying evidence across lineages rather than repeatedly reframing residual errors.Source: §3.2 · p. 7
  • On go emotions, CoCoEvolve tests SQL drafts on a cheap sample and repairs syntax failures within the mutation, while broken GEPA variants consume up to 71% of the budget.This turns compiler errors into actionable context before outer-loop evaluation.Source: §3.2 · p. 7

3.3 DATA ENGINEERING

  • 77.4% test solve rate on data-eng-bench surpasses GEPA at 73.6% and Meta-Harness and AdaEvolve at 71.7%, up from the 59.1% baseline.On dev, CoCoEvolve reaches 68.0%, tying Meta-Harness for the highest solve rate.Source: §3.3 · p. 8, Table 3 · p. 8
  • In data engineering, the mutator retrieved and re-implemented an archived skill after a later hillclimb had dropped it, matching a returning failure to its prior resolution.The ledger keeps capabilities visible even when they are absent from the current parent’s trace.Source: §3.3 · p. 8

3.4 MATHEMATICAL, SYSTEMS, AND PROMPT OPTIMIZATION

  • CoCoEvolve is best or tied best in 25 of 44 mathematical and ADRS Avg/Best cells; on Signal Processing it scores 0.766 versus AdaEvolve’s 0.747.The advantage is largest on plateau-prone tasks, while gaps largely vanish on saturated objectives and mixed-backbone results show backbone dependence.Source: §3.4 · p. 8, Table 4 · p. 9
  • On HotpotQA, CoCoEvolve reaches 0.793 held-out accuracy versus GEPA’s 0.683 and the seed’s 0.363; AIME shows no reliable separation amid repeated-evaluation variability.CoCoEvolve also leads GEPA and the seed on every IFBench split, though its held-out margin is narrower.Source: §3.4 · p. 9, Table 5 · p. 9

4 RELATED WORK

Related work spans LLM-guided evolutionary search, context curation, and known vulnerabilities of evolutionary approaches. CoCoEvolve is distinguished by changing the mutation interface to use curated history and live-environment interaction.

Evolutionary Harnesses

  • Evolutionary harnesses use LLMs as semantic mutation operators, building on quality-diversity search and extending from algorithm discovery to prompt and compound-system optimization.Examples include FunSearch, AlphaEvolve, and OpenEvolve (Novikov et al., 2025), while GEPA uses language-valued execution traces (Agrawal et al., 2026b).Source: §4 · p. 9
  • Most related methods retain parent-local, one-shot mutation, whereas CoCoEvolve changes the interface through meta-reflection over evolutionary history and the target environment.Meta-Harness uses a coding-agent proposer with filesystem access to prior traces; CoCoEvolve combines curated context, structured history queries, and a live environment.Source: §4 · p. 10

Context Curation Approaches

  • Context-curation work evolves playbooks or curation skills, while CoCoEvolve distills the optimization trajectory itself into a compact ledger.The cited approaches include evolving contexts as playbooks (Zhang et al., 2026) and co-evolving curation skills alongside artifacts (Ye et al., 2026).Source: §4 · p. 10

Limitations of Evolutionary Approaches

  • Prior work identifies hidden regressions, continual-learning overfitting, and archive context costs as vulnerabilities of evolutionary frameworks.CoCoEvolve addresses these with bounded, queryable access to historical co-regressions.Source: §4 · p. 10

5 CONCLUSION

The conclusion presents CoCoEvolve as an evolutionary optimizer for production AI systems that extends parent-local reflection through historical evidence retrieval and live-environment probing. The appendices document the optimizer, experiments, and evolved artifacts; the authors also disclose their use of AI tools.

5 CONCLUSION

  • CoCoEvolve combines curated historical evidence, on-demand tools for probing the live environment and evolutionary history, and a coding-agent mutator to reason over both.The authors say results across domains suggest this approach offers an effective path to improving prompts, code, tools, data models, and pipelines of real-world AI systems.Source: §5 · p. 10

A TABLE OF CONTENTS

  • The appendices cover the evolutionary loop, tools, ablations, prompts, benchmark protocols, detailed analyses, and complete base and final programs for three tasks.Appendix K contains programs for Heilbronn Convex, Cloudcast, and TXN.Source: A TABLE OF CONTENTS · p. 13

B USE OF LARGE LANGUAGE MODELS (LLMS) AND OTHER AI TOOLS

  • The authors report using LLMs for literature discovery and prose polishing, and coding assistants to help implement their ideas.They state that they reviewed and verified suggested references, claims, and revisions and take responsibility for the final content.Source: B USE OF LARGE LANGUAGE MODELS (LLMS)… · p. 14

C ALGORITHM AND METHOD

CoCoEvolve combines curated evolutionary history with task-environment interaction in an iterative loop, then uses task-specific mechanisms for targeting, population coverage, and parent selection. The appendix also specifies the experimental controls for its ablations.

C.1 EVOLUTIONARY LOOP

  • Each iteration selects a parent, curates context, augments the environment with history queries, lets Cortex Code investigate and validate a child, then updates the archive and population.The loop’s preparation mechanisms are layered according to each domain’s search objective.Source: §C.1 · p. 14

C.2 ON-DEMAND HISTORY AND ENVIRONMENT TOOLS

  • The on-demand tools expose archive lineage and trajectory queries, while DABStep additionally provides SQL, scratch-agent deployment, and target-question pre-screening.The archive need not be placed fully in the prompt: tool results enter context only when requested.Source: §C.2 · pp. 14–15, Table 6 · p. 15

C.3 CURATED OBSERVATIONS: KERNEL, DECAY, AND SOFTMAX

  • For DABStep, similarity-weighted estimates guide targeting; softmax samples unsolved instances by gain, while repeated failures decay unless an archive solve resets the penalty.The estimator emphasizes nearby tested instances, and estimates for untested coordinates are capped at 0.7.Source: §C.3 · p. 15

C.4 HYPERPARAMETERS

  • Default math, ADRS, and DABStep runs use MAP-Elites parent draws; InfoGain targeting is enabled only for DABStep, while AI-function runs use the ℓp roster.Signal Processing exposes per-task queries but does not use InfoGain targeting.Source: §C.4 · p. 15

C.5 BUDGETED POPULATION ℓp OBJECTIVE AND PARENT SELECTION

  • The capped active roster greedily retains programs only while they add uncovered dimensions, while the ℓp score rewards coverage and, for p > 1, consolidation.Greedy construction preserves complementary capabilities rather than selecting solely by aggregate fitness.Source: §C.5 · p. 16
  • For population objectives, the loop directly optimizes the roster’s ℓp score; for scalar objectives, the same roster can instead guide parent selection without replacing the evaluator.At p = 2, parent sampling emphasizes larger complementary contributions quadratically.Source: §C.5 · p. 17

C.5.1 RARE-CAPABILITY CARRIER SELECTION

  • Rare-capability selection weights dimensions by population solve rate and favors fit carriers of rare capabilities, increasing preference for lineages that accumulate multiple hard capabilities.The mechanism changes which existing program is selected, not the evaluation pool or donor set.Source: §C.5.1 · p. 17

D ABLATION DETAILS

  • The ablations isolate parent selection, mutation execution and self-validation, history access, and similarity-manifold targeting; each claim-driven comparison holds the rest of the loop fixed.The controls match model, budget, seed, split, parent policy, artifact surface, iteration count, and evaluation allowance unless a subsection states an exception.Source: D ABLATION DETAILS · pp. 17–18

D.1 PARENT SELECTION

The parent-selection experiments compare four policies under matched search setups and separately vary the ℓp roster exponent on HotpotQA. The ℓp roster leads the observed held-out scores, while the exponent sweep is nonmonotonic.

Setup

  • The four policies differ in parent eligibility: select-best, MAP-Elites, per-instance Pareto-frontier sampling, and an ℓp roster sampled by marginal coverage.Within each task, the arms share the seed, split, backbone, mutation process, island configuration, and 100-iteration budget.Source: §D.1 · p. 18
  • The HotpotQA exponent sweep compares p ∈ {0.5, 1, 1.5, 2} with the roster construction held fixed.Each setting uses one 100-iteration search, and p is the only treatment varied.Source: §D.1 · p. 18

Results

  • The ℓp roster has the highest observed held-out score on both HotpotQA and IFBench.The comparison uses one 100-iteration search per arm and single held-out test evaluations.Source: Table 8 · p. 18, §D.1 · p. 19

Analysis

  • p = 2 leads HotpotQA validation and held-out test, while the p = 1.5 dip shows the sweep is not monotonic.The sweep varies p while keeping roster construction fixed.Source: Table 9 · p. 18, §D.1 · p. 19

D.2 MUTATION EXECUTION AND SELF-VALIDATION

The ablation shows monotonically better scores as the mutator gains agentic editing and then pre-commit testing. These capabilities help preserve existing functionality and are especially useful when changes affect executable code or dependent objects.

Setup

  • A1 is a static completion, A2 can edit and query history but cannot execute the edited candidate, and A3 adds pre-commit execution or deployment and testing.On DABStep, A2 can query the database but cannot deploy the edited agent; A3 can query, deploy, and test it.Source: §D.2 · p. 19

Results

  • The Heilbronn Convex summary includes one trial stopped at 83 iterations, and the DABStep A3 result reuses the published champion rather than a new run.DABStep values are fresh full-pool re-evaluations; one 100-iteration trial is reported per arm.Source: §D.2 · p. 19
  • DABStep test accuracy rises from 47/150 (A1) to 98/150 (A2) to 137/150 (A3), while Heilbronn Convex mean score rises from 0.0260 to 0.0270 to 0.03066.Table 10 summarizes three trials per Heilbronn Convex arm and one full-pool champion evaluation per DABStep arm; deltas are relative to A1.Source: Table 10 · p. 19, §D.2 · p. 20

Analysis

  • On Heilbronn Convex, A1 frequently replaces solver portions, with roughly one quarter of scored edits regressing in a matched trial; A2 regressions fall to about one in eight and A3 to roughly 4%.A2 generally preserves the solver and adds components; A3 tests changes through scratch execution.Source: §D.2 · p. 20
  • On DABStep, database-grounded editing raises train/test accuracy from 33.7%/31.3% to 67.3%/65.3%, while deployment and end-to-end testing enable 95.3%/91.3%.A2 corrects fee-rule enumeration and tightens the exact-match response contract; A3 validates a structural replacement for the fee-computation procedure.Source: §D.2 · p. 20

D.3 HISTORY-ACCESS INTERFACE

This section describes the history-access interface for optimization, reporting ablations of history access and similarity-manifold targeting alongside prompts and constraints for the data-pipeline workflow and optimization-strategy selection.

D.3 HISTORY-ACCESS INTERFACE

  • On DABStep, B4 scored 286 train and 137 test, versus B1’s 234 train and 116 test; on Signal Processing, B4 scored 0.7681, above B3 (0.7487), B2 (0.7074), and B1 (0.6614).In the four primary DABStep runs, B2 scored 210 train and 99 test, while B3 scored 242 train and 119 test. The Signal Processing ablation disabled the reflector/proposer; B4’s 0.7681 was near the published Sonnet-5 mean of 0.7656 (best 0.7841).Source: §D.3 · pp. 20–21, Table 11 · p. 20, Table 12 · p. 21
  • History tools helped the mutator inspect prior programs, while ledger-only access could expose uncertain projected scores without the programs or traces needed to assess them.A useful merchant-instruction repair was rejected in B2 because its assigned target question was unrelated to the bug, while B1’s repair improved its target and survived the evaluation gate. A second B4 run scored 248/300 train and 114/150 test, versus the primary run’s 286/300 and 137/150; tool-use patterns differed, but tool use was observed rather than experimentally assigned.Source: §D.3 · pp. 21–22

D.4 SIMILARITY-MANIFOLD TARGETING

  • 286/300 train and 137/150 test: C3 outscored C1 (95/300, 42/150) and C2 (113/300, 51/150) on DABStep.C3 gained 189 questions and regressed 4, versus 29 gained/35 regressed for C1 and 28 gained/16 regressed for C2; 143/189 C3 gains transferred beyond its directly targeted questions. Target selection and untested-question expansion varied together, so their individual contributions were not isolated.Source: §D.4 · p. 23, Table 13 · p. 23, Table 14 · p. 23

E PROMPTS USED

  • The prompts equip mutators with a live-updated ledger, on-demand history queries, and task workspaces, and instruct DABStep agents to test the existing tool before diagnosing a failure.For data engineering, CoCoEvolve evolves a knowledge-graph construction pipeline that explores the warehouse and produces KG.md for a separate fixed coding agent; prompt guidance supports changes to agent topology and reusable skills based on evaluation evidence.Source: E PROMPTS USED · pp. 23–26
  • The workflow runs a designed KG pipeline, a fixed platform compiler that produces adherence tests, and a fixed Harbor agent that builds dbt models and self-tests them.The Harbor agent’s iteration is evaluated by a pytest score.Source: E PROMPTS USED · p. 26
  • The pipeline artifact is a valid Python dict edited only within the evolve-block markers, using generic templates, placeholders, metadata-only skills, and a KG capped at 300 lines without solution SQL.Agent prompts retain {task_name}, {db_name}, and {timeout_sec}; skill bodies are assigned by agents in .md files.Source: E PROMPTS USED · p. 26
  • A strategy-pool controller assigns a freed worker to an active strategy, a newly proposed strategy, or plain hillclimbing, while an advisor proposes at least one high-level idea distinct from prior attempts.The controller chooses the action but does not design strategies or edit code; the advisor returns ideas in a JSON object with an ideas array.Source: E PROMPTS USED · pp. 26–27

F.1 DETAILED METHODOLOGY

The detailed methodology describes how CoCoEvolve, Meta-Harness, GEPA, and no-search baselines were evaluated on DABStep. It also clarifies that the combined-suite leaderboard table ranks by hard-question accuracy and combines training and test questions for the paper's search-method rows.

F.1 DETAILED METHODOLOGY

  • The CoCoEvolve and GEPA rows use a 100-iteration budget on the same seed, while Meta-Harness uses a faithful Terminal-Bench-2 loop port with Cortex as proposer and the pooled-parallel DABStep evaluator.Meta-Harness's main-paper row uses a self-test variant; its held-out champion scores 78.4% hard accuracy and 78.7% total accuracy.Source: §F.1 · p. 27
  • The two no-search baselines distinguish the unchanged seed specification from a general-purpose coding agent given documentation and read-only live-table access but no seed tools or instructions.The seed row measures what the unmodified seed solves; the stock agent runs a fresh session per question without memory or an evolutionary loop.Source: §F.1 · p. 27

F.2 COMBINED-SUITE LEADERBOARD

  • Table 15 places the paper's search-method and no-search evaluations beside public leaderboard entries; the table ranks by hard-question accuracy, while public submissions report combined-suite accuracy without separate train/test breakdowns.The paper's CoCoEvolve, Meta-Harness, GEPA, and no-search rows combine 300 training and 150 test questions.Source: §F.2 · p. 27

F.3 WHY SELF-TEST TOOLS DID NOT PREVENT META-HARNESS’S CHURN

Across these appendix sections, CoCoEvolve recovers a hidden rule system and evolves a DABStep agent around a deterministic compute kernel. Meta-Harness’s self-test tools did not prevent a prolonged stall involving repeated dead-end queries and a semantic-view mismatch.

F.3 WHY SELF-TEST TOOLS DID NOT PREVENT META-HARNESS’S CHURN

  • After iteration 53, Meta-Harness made no further gains in 47 iterations; traces show repeated nonexistent-table searches and semantic-view edits that passed preliminary checks but failed deployment.The authors caution that these traces explain failures in this run, not necessarily every use of the self-test tools.Source: §F.3 · pp. 27–28
  • At iteration 61, separate edits to the semantic view’s SQL clause and JSON caused their representations to drift, while check deploy checked compilation rather than cross-references.The live Snowflake compiler exposed the mismatch, returning only a line and position rather than a diff.Source: §F.3 · p. 29
  • The observed failures suggest surfacing question-data locations and retaining learned dead ends, alongside structural validation of both semantic-view representations.In this run, self-test access alone supplied neither safeguard.Source: §F.3 · p. 29

F.4 EXAMPLES OF BEST PROGRAMS

  • The DABStep champion replaced a 5KB SQL procedure with a 53KB, 14-mode Python engine; it scored 293/300 in-run and 286/300 on fresh re-evaluation.The authors highlight demoting text-to-SQL in favor of a deterministic compute kernel as the architectural shift they believe contributed most to performance.Source: §F.4 · p. 29

G.1 SYNTHETIC DIAGNOSTIC: LOCKPICKING WITH INTERFERENCE

  • The synthetic diagnostic uses 50 labeled strings and 50 mutation iterations, with no held-out split because it measures reconstruction of a latent rule system.Its aggregate score can leave structurally different rule tables indistinguishable across many examples.Source: §G.1 · pp. 31–32
  • CoCoEvolve reaches 50/50 after six mutations, versus OpenEvolve’s 27/50 and AdaEvolve’s 45/50 plateau on Lockpicking with Interference.AdaEvolve plateaued at iteration 34; CoCoEvolve recovered the latent rule system while the baselines fit surrogate rules.Source: §G.1 · p. 32

G.2 DETAILED METHODOLOGY

  • AI-function candidates remain executable SQL while evolution may rewrite instructions, schemas, and pipelines; live-task test rows stay hidden until post-run scoring.The search pool is partitioned once, with validation used for fitness and acceptance and a disjoint test table withheld during evolution.Source: §G.2 · p. 32
  • A curator selects a parent and up to ten representative rows; the mutator may verify SQL on them up to three times, while full validation is the sole admission gate.Runs use 50 mutation attempts, one island, population 40, archive capacity 20, and eight parallel workers.Source: §G.2 · pp. 32–33

G.3 EXAMPLES OF BEST PROGRAMS

The best-program examples show evolution turning broad or generic starting instructions into operational procedures with explicit precedence, exception, classification, arithmetic, token-normalization, and serialization rules.

G.3 EXAMPLES OF BEST PROGRAMS

  • The lockpicking examples contrast CoCoEvolve’s deterministic scoring expression with AdaEvolve’s complete 45/50 prompt; one worked example produces an output of 6.In the deployed AI-function body, the deterministic integer is passed to AI COMPLETE only for schema-conforming serialization.Source: §G.3 · pp. 33–35
  • For GoEmotions, failure analysis exposed over-labeling; a conservative default and added rules passed all ten selected live-verifier rows and raised the objective from 0.0022 to 0.1741.The final champion uses strict-priority rules, explicit exceptions, a closed vocabulary, and structured output.Source: §G.3 · p. 37
  • Held-out scores were 0.444 (GoEmotions), 0.813 (finqa), 0.953 (convfinqa), 0.587 (multilexnorm; +7.7 pp over GEPA), and 0.737 (lex glue).The GoEmotions champion was discovered at iteration 47; the multilexnorm score is a 7.7 pp gain over GEPA.Source: §G.3 · pp. 37, 39, 40
  • The finqa and convfinqa champions specify reference resolution, formula selection, sign direction, units, and serialization, including dialogue-history rules for conversational questions.The finqa program distinguishes percent change, signed differences, ratios, averages, CAGR, and yes/no questions; convfinqa resolves references and reuses prior answers without silently substituting a nearby year.Source: §G.3 · pp. 38–39
  • The multilexnorm champion transforms only high-confidence tokens while preserving capitalization, spacing, punctuation, mentions, hashtags, URLs, proper nouns, and uncertain or non-English tokens.It also records exceptions for contractions, elongated forms, and truncated -ing words.Source: §G.3 · p. 39
  • The lex glue champion expands a short glossary into a hierarchy of issue-area distinctions using priority scans, exceptions, and counterexamples to separate substantive holdings from procedural posture.Examples distinguish First Amendment from Privacy, Judicial Power from Economic Activity, and Federalism from Unions.Source: §G.3 · pp. 39–40
  • Programs prioritize attorney-fee holdings as Attorneys, distinguish speaker exclusion (First Amendment) from captive listeners (Privacy), separate agency review (Judicial Power) from scope (Economic Activity or Unions), and classify labor preemption as Federalism.For substantive labor rights, the classification is Unions; the underlying organizing or bargaining right remains distinct from labor preemption.Source: §G.3 · p. 40
  • Across scales, evolution replaces broad task descriptions with explicit precedence, exception, and serialization rules.Source: §G.3 · p. 40

H.1 DETAILED METHODOLOGY

This section details CoCoEvolve’s architect-led workflow for evolving a data-engineering pipeline that builds task-specific knowledge graphs and optional executable feedback, including its use of live warehouse inspection, worked examples, and documented environment and skill guidance. It also describes how the pipeline is configured and evaluated on data-eng-bench.

H.1 DETAILED METHODOLOGY

  • The pipeline produces a task-specific, 300-line-limited markdown knowledge graph covering verified warehouse structure, lineage, business rules, metrics, and task-specific pitfalls.The authoring process checks physical table locations, column types, categorical values, null encodings, and row counts using warehouse queries rather than relying on a potentially stale sources.yml file.Source: §H.1 · p. 40
  • The evolved Python PIPELINE SPEC jointly controls knowledge-graph authoring protocols, agent orchestration, and the protocol for authoring executable adherence tests.The specification produces KG.md and test_kg.py; the tests are derived from the graph and provide solver feedback during compilation.Source: §H.1 · p. 41
  • At inference, evaluated agent roles share the claude-sonnet-5 backbone and harness, while a fixed claude-opus-4-8 author generates adherence tests that were not used in every run.The generated tests provide self-correction feedback and are not the hidden grader; adherence testing is therefore optional additional context, not a uniform component of reported results.Source: §H.1 · p. 41
  • CoCoEvolve uses the best program after 200 iterations, with one island, 15 active candidates, archive capacity 8, and 8 parallel evaluations; comparisons use matched backbones, agents, budgets, and seed programs.The benchmark has 50 randomly sampled development tasks and 53 held-out test tasks from 103 total tasks.Source: §H.1 · pp. 40–41

H.2.1 EVOLVED PROGRAMS

  • The evolved specification retains three seed skills and adds four, including warehouse preflight, persistent dbt profile-path guidance, column verification, and worked examples for weighted or ratio-based metrics.It also tightens adherence tests to recompute stated business rules against source data and disallows negative-only assertions; the example retains a sole architect agent.Source: §H.2.1 · pp. 41, 42, 44
  • Use one architect for warehouse exploration and KG writing in the same session; split agents only if coverage shows exploration was skipped and a handoff would help.The KG must follow every listed skill, with each authored under runtime_skills/<name>.md.Source: §H.2.1 · pp. 46–47
  • The architect must query the live warehouse, verify each named table and its columns, and write a Knowledge Graph of confirmed facts rather than solution SQL.The KG is limited to 300 lines.Source: §H.2.1 · p. 46
  • Before schema exploration, confirm that the session has an authorized, usable warehouse and verify dbt profiles-directory resolution for new projects.Include the confirmed warehouse fix and profiles-directory fact in KG.md’s Agent Usage Guide.Source: §H.2.1 · p. 47
  • For weighted-composite or percentile/ratio metrics, compute a worked numeric example from live rows and state unresolved ambiguity rather than presenting an unsupported interpretation as fact.The instructions call for following the stated formula, including cap, inversion, boundary behavior, and NULL-fallback substitution.Source: §H.2.1 · pp. 46–47

H.2.2 FINAL PIPELINE SPECIFICATION ARTIFACTS

The final pipeline specification artifacts give a downstream solver verified warehouse facts, task rules, and executable checks for a marketing attribution pipeline. They cover source assumptions and attribution rules, output grain and aggregation hazards, conversion metrics, channel defaults, data-quality requirements, and safeguards against revenue inflation and unsupported conversions.

H.2.2 FINAL PIPELINE SPECIFICATION ARTIFACTS

  • The KG records confirmed source tables and columns, mapping rules for NULL UTM fields, and verified environment and schema settings for the downstream model.It specifies conversion_rate as conversions divided by sessions with a divide-by-zero guard and four-decimal rounding.Source: §H.2.2 · pp. 47, 49, 50
  • The joined view lacks T2_EVENT_VALUE, so this environment requires the specified revenue fallback to STG_GA__EVENTS.EVENT_VALUE; session-to-event fanout is about 3.1x.The view has 15 columns and 15,775 rows for 5,053 distinct sessions; revenue should be summed per session before aggregation to avoid fanout inflation.Source: §H.2.2 · pp. 48–49
  • The specification anchors its 90-day window to the maximum session date, 2026-01-03, and warns that event timestamps are VARCHAR values with 3,801 of 15,047 failing timestamp casts.The required output grain is one row per attribution_date, channel, medium, and campaign; session timestamps also require casting, and 46 session dates are pre-2020 outliers.Source: §H.2.2 · pp. 48–50
  • The live data conflicts with the literal conversion rules: IS_CONVERTED = '1' counts 47 sessions, while the truthy-token interpretation counts 160, and no purchase event is present.A literal purchase-event rule would yield zero attributed revenue; the artifact flags both issues without resolving them for the solver.Source: §H.2.2 · pp. 49, 51
  • The accompanying test suite turns KG statements into executable checks against warehouse facts and output business rules, with failures pointing to the relevant KG statement.The tests are derived from the KG rather than the instruction file, solution SQL, or hidden grader.Source: §H.2.2 · pp. 51–52
  • The joined source has 15,775 rows versus 5,053 distinct sessions, a roughly 3.1x fanout that requires summing revenue per session before final aggregation.The specification also states that `t2_event_value` is absent from the joined view, so revenue falls back to `stg_ga__events.EVENT_VALUE`.Source: §H.2.2 · p. 54
  • Source anchors check that truthy `is_converted` tokens match 160 sessions, literal `1` matches 47, and falsy tokens match 4,893.The checks also flag that `event_name = 'purchase'` returns 0 rows, creating a structural risk that attributed_revenue is zero for every output row.Source: §H.2.2 · pp. 54–55
  • The required output is a populated dbt model with one unique row per (attribution_date, channel, medium, campaign), including sessions, conversions, conversion_rate, and attributed_revenue.The revenue field is derived from `t2_event_value` or the fallback event value, summed per session first.Source: §H.2.2 · pp. 56–57
  • Output requirements forbid NULLs in required columns, require at least one grain row, define conversion_rate as conversions / sessions, treat converted sessions as a subset of all sessions, and require sessions to be a nonnegative count of distinct sessions.Source: §H.2.2 · pp. 58–59
  • The tests require conversion_rate to equal conversions divided by sessions, rounded to four decimals, and to remain defined when sessions are zero.They also check that conversions are non-negative.Source: §H.2.2 · pp. 59–60
  • The mart’s SUM(sessions) must match distinct source sessions in the 90-day trailing window, and every attribution_date must fall within that window.The window is anchored to the maximum observed session_start date.Source: §H.2.2 · p. 60
  • Output channel, medium, and campaign values must not be NULL; the specified defaults are direct for UTM_SOURCE and none for UTM_MEDIUM and UTM_CAMPAIGN.Source: §H.2.2 · p. 61
  • The revenue test guards against session-to-event fanout inflation by requiring total attributed_revenue not to exceed the bound from per-session summed EVENT_VALUE.The artifact identifies approximately 1:3.1 session-to-event-row fanout.Source: §H.2.2 · p. 62
  • Because the source has no literal purchase events, the spec-faithful purchase rule requires attributed_revenue to be zero across output rows.Source: §H.2.2 · pp. 62–63
  • The conversion-count test caps SUM(conversions) at 160, the upper bound from the specified truthy is_converted values within the 90-day window.Source: §H.2.2 · p. 63

I.1 DETAILED METHODOLOGY

Math and ADRS comparisons use matched backbones and shared 100-iteration budgets, with Human/SOTA references defined by benchmark type. The appendix details higher-precision results and finds that Signal Processing leaders share a filtering strategy but differ in regularization.

I.1 DETAILED METHODOLOGY

  • Math and ADRS runs use matched backbones and 100-iteration budgets, with results reporting the best objective reached within that shared budget.The AdaEvolve control loop is rerun with CoCoEvolve’s backbone to control for model choice.Source: §I.1 · p. 63
  • Table 17 records each task’s instance size, reported scalar, and Human/SOTA reference; math references take the better of published human and automated results, while systems use AdaEvolve references.The math automated references are drawn from AlphaEvolve and AdaEvolve.Source: §I.1 · p. 63, Table 17 · p. 64

I.2 HIGHER-PRECISION MATH AND ADRS TABLES

  • Tables 18 and 19 provide greater-precision values corresponding to Table 4, while retaining rankings computed from the underlying values.Table 18 reports mathematical benchmarks; Table 19 reports ADRS systems benchmarks.Source: §I.2 · p. 64, Table 18 · p. 64, Table 19 · p. 64
  • Signal Processing’s overall score blends a composite J(θ) with smoothness, correlation, noise reduction, and success rate.J(θ) combines slope changes, lag, error, and false reversals.Source: §I.2 · p. 64
  • CoCoEvolve and AdaEvolve converge on the same broad Signal Processing filter, while swapping their regularization rules also swaps measured performance.CoCoEvolve scales a fixed coefficient by a robust noise estimate; AdaEvolve uses a clipped noise-to-motion rule.Source: §I.2 · p. 65

I.3 PER-TASK ANALYSIS

CoCoEvolve leads or ties more math and ADRS Avg/Best cells overall than the other evaluated methods, but results vary by backbone and objective. The selected traces illustrate plateau escape, structural rewrites, and saturation rather than explaining every aggregate result.

I.3 PER-TASK ANALYSIS

  • 25 of 44 Avg/Best cells are led or tied by CoCoEvolve, compared with 18 for GEPA, 13 for AdaEvolve, and 6 for OpenEvolve.CoCoEvolve leads or ties 13 of 24 math cells and 12 of 20 ADRS cells across both backbones.Source: §I.3 · p. 65
  • On Sonnet-5, CoCoEvolve has the highest Heilbronn Convex mean and ties GEPA on Best; ADRS leaders vary by task, with AdaEvolve and GEPA leading some cells.CoCoEvolve leads both Avg and Best on Cloudcast, EPLB, and TXN, and LLM-SQL on Avg; AdaEvolve leads Prism and GEPA leads LLM-SQL on Best.Source: §I.3 · p. 65
  • On GPT-5.6-terra, all four methods lead at least one listed objective, and several saturated objectives have ties near reference values.The passage notes that means are more informative than Best values for Circle Packing instances and MinMax Distance.Source: §I.3 · p. 65
  • The traces examine stalled-trial recovery on Heilbronn Convex, a structural rewrite on TXN, and tied final scores on saturated Circle Packing; they are selected-run explanations, not aggregate results.InfoGain is inactive because these tasks have no instance pool.Source: §I.3 · p. 65
  • Figures 4–5 show Heilbronn Convex layouts and best-so-far traces, while Figures 6–7 show Circle Packing (Square) layouts and traces.The listed figures are in Appendix I.3.1 and I.3.2, respectively.Source: §I.3 · p. 65

I.3.1 HEILBRONN CONVEX (N=13)

Across four task analyses, CoCoEvolve’s results reflect distinct behaviors: recovering stalled Heilbronn trials, reaching a saturated Circle Packing ceiling, constructing a Cloudcast solution, and replacing the TXN seed heuristic. These traces explain selected runs rather than ranking every method in every setting.

I.3.1 HEILBRONN CONVEX (N=13)

  • 0.03066 ± 0.00036 is CoCoEvolve’s three-run Heilbronn Convex mean, with finals of 0.03094, 0.03078, and 0.03025 despite two trials spending most of the budget near 0.028.Both GEPA and CoCoEvolve reach 0.03094 Best, so the analysis concerns CoCoEvolve’s higher Avg rather than a unique Best champion.Source: §I.3.1 · p. 65

I.3.2 CIRCLE PACKING (SQUARE)

  • All methods reach the 2.63598 Circle Packing ceiling in at least one Sonnet-5 trial, so the trace compares time to ceiling rather than final scores.GEPA reaches it at iteration 8, CoCoEvolve at 24, AdaEvolve at 91, and OpenEvolve remains at 2.6298 at iteration 100, reaching the ceiling at iteration 111.Source: §I.3.2 · pp. 66–67, Figure 7 · p. 67

I.3.3 CLOUDCAST

  • The Cloudcast champion evaluates six tree families beyond hub search, and scratch candidates tested against the live evaluator help reduce cost from 974 to 636 in the first two iterations.Later Dreyfus–Wagner ranking and TSP-relay construction produce the remaining improvement; GEPA, AdaEvolve, and CoCoEvolve finish in a tight 618 cluster.Source: §I.3.3 · pp. 67–68

I.3.4 TXN SCHEDULING

  • The TXN champion replaces tabu search with assignment/greedy schedule construction followed by simulated annealing and local-search cleanup, improving both Avg and Best beyond the other methods.The analyzed CoCoEvolve Best-cell run scores 4,348 against a seed and Human/SOTA value of 2,725.Source: §I.3.4 · p. 68

J PROMPT OPTIMIZATION

This section describes optimizer variants and workload-specific methods for HotpotQA, Heilbronn Convex, broadcast routing, and transaction scheduling, including evaluation comparisons and evolved programs.

J.1 DETAILED METHODOLOGY

  • The GEPA comparison evaluates AIME, HotpotQA, and IFBench from the same initial program, with 100 iterations per CoCoEvolve run.For HotpotQA, the four-stage prompt pipeline uses GEPA’s 150/300/300 train, validation, and held-out-test split, and frozen Sonnet-5 as mutator and student.Source: §J.1 · p. 68
  • Repeated AIME and IFBench evaluations measure evaluation variability rather than search variability: each optimizer produced one candidate, which was evaluated three times per split.On AIME test, correctness changed across repeats for five baseline, eight GEPA, and six CoCoEvolve questions, although the baseline aggregate score remained 25/30 each time.Source: §J.1 · p. 69

J.2 AN EFFICIENCY VARIANT: SIM-GRAPH TARGETING AND CASCADED EVALUATION ON HOTPOTQA

  • The HotpotQA variant combines similarity-based target selection, P0/P1/P2 cascaded evaluation, and mutator self-testing, so its results cannot isolate any one mechanism.Each mutation receives one focus question; the cascade evaluates at most 50 of 150 training questions per iteration, excluding bootstrap, self-tests, and separately sampled validation passes.Source: §J.2 · pp. 69–70
  • The combined HotpotQA variant matches full-pool train accuracy (0.800), has 0.007 higher validation accuracy, and has 0.026 lower test accuracy, about eight questions out of 300.With one search and one final re-score per variant, the result does not establish equal held-out quality or identify which added mechanism caused the differences.Source: Table 20 · p. 70, §J.2 · p. 70
  • Estimated evaluator time falls about 4–5×, from 25–30 hours for full-pool evaluation to 5.5–6 hours for the variant.These are estimates extrapolated from observed timing, not measured total run times; median variant time was 121 seconds per iteration versus roughly 900–1100 seconds for a full 450-question pass.Source: §J.2 · p. 70

K COMPLETE BASE AND FINAL PROGRAMS

K.1 HEILBRONN CONVEX

  • The Heilbronn Convex final program uses deterministic multi-restart CMA-ES, exact-bilinear trust-region SLP polishing, and iterated local search to maximize the minimum-triangle-area-to-hull-area ratio.It alternates jittered 13-gon and two-ring restart families and re-scores candidates against a safe regular-13-gon fallback.Source: §K.1 · pp. 74–76

K.2 CLOUDCAST

  • The Cloudcast listing shows a routing algorithm that computes weighted shortest paths and evaluates single-hub plans using actual edge costs and throughput.The displayed code removes unusable edges before computing paths and carries the original edge data into constructed paths.Source: §K.2 · p. 77
  • The router also searches one- and two-hub broadcast plans, then applies an arborescence fallback and direct shortest paths for destinations still unassigned.The fallback uses a minimum spanning arborescence over terminal-to-terminal costs when possible.Source: §K.2 · pp. 78–79
  • The Cloudcast router ranks complete broadcast assignments by replaying egress cost and throttle-aware instance-time cost, using fixed harness settings and a data-volume-invariant proxy.The proxy models proportional ingress/egress throttling, which can make a tree with cheaper edge costs more expensive overall.Source: §K.2 · pp. 80–81
  • It compares terminal-only metric-closure arborescence, unrestricted-relay, and node-split capacity-aware min-cost-flow candidates, patching unmet flow destinations with shortest paths.Each candidate is evaluated with the same proxy, so the router selects the cheapest complete assignment.Source: §K.2 · pp. 80, 82, 83, 87
  • The Dreyfus–Wagner DP preserves minimum egress cost with a composite edge weight, then uses throughput to favor ties with fewer relay hops.The implementation uses a bitmask DP over destination subsets and nodes, and is skipped when dependencies, reachability, or terminal-count limits prevent its use.Source: §K.2 · pp. 83–84
  • A current-flow candidate uses throughput-weighted effective resistance to choose among equal-cost paths, while a TSP relay chain avoids fan-out throttling at the possible cost of longer paths.Both are scored alongside the other candidates before selection.Source: §K.2 · pp. 85–87

K.3 TXN

  • The TXN scheduler initializes from Copeland, multidimensional scaling, assignment-problem, and multi-start greedy sequences, then runs tabu search with swap and Or-opt-1 moves.It uses aspiration, double-bridge diversification and restarts, and final swap local-search cleanup.Source: §K.3 · pp. 87–90
  • Pairwise transaction costs are computed in both orders to form a cost matrix and Copeland wins, while assignment uses cost magnitudes rather than win counts.The implementation derives both directional costs from each transaction pair and ranks Copeland candidates by their wins.Source: §K.3 · p. 90
  • An assignment relaxation produces directed cycles that are stitched by greedy insertion in either orientation; the optimizer falls back to Copeland order when SciPy is unavailable or n < 2.The code evaluates the resulting assignment-based sequence against a greedy schedule seeded by the top Copeland candidate.Source: §K.3 · pp. 90–91
  • The optimizer compares the assignment-based schedule with a greedy extension from the top Copeland seed and starts refinement from the lower-cost schedule.This comparison retains whichever of the two candidate schedules has the lower measured sequence cost.Source: §K.3 · p. 91
  • Simulated annealing cools from a temperature estimated using sampled cost changes and explores schedules through swaps, single-transaction relocations, and two-transaction block relocations.Moves may be accepted despite increasing cost according to the temperature-dependent probability rule.Source: §K.3 · pp. 91–92
  • When annealing stalls at low temperature, reheating alternates between double-bridge perturbations of the best schedule and resets to that best schedule, up to six times.A final safeguard resets the current schedule when its cost exceeds the best by more than 3% at very low temperature.Source: §K.3 · p. 91
  • First-improvement cleanup repeatedly applies swaps and Or-opt relocations of block lengths 1, 2, and 3 until no improvement remains or the time limit is reached.The routine returns the best cost and corresponding schedule found.Source: §K.3 · pp. 92–93

Related papers

Loading 2610.12794v1…