Source-linked AI summary
AgenticEvolve: Evolutionary Program Search with Persistent Language Model Agents that Design and Run Experiments
Ze-Yu Peng, Qing-Yu Lan, Zhan-He Wang, Jun Zhang, Yun-Song Piao
TL;DR
LLM-based evolutionary program search typically generates candidates from parent programs and scores, with empirical feedback supplied through formal evaluations. AgenticEvolve assigns persistent agents to search islands and run experiments between evaluations, and its experiments improve heuristic quality and equation recovery across the reported tasks.
Problem
In LLM-based evolutionary search, empirical information about program behavior enters through formal evaluations, which consume budget and often use instances hidden from the model.
Source: §I · p. 1Method
AgenticEvolve runs a persistent LLM agent on each evolutionary island, where it edits candidates, runs self-designed experiments between commits, and submits programs for formal evaluation.
Source: §I · p. 1, B. One Island: Persistent Conversation… · p. 3Results
Across two heuristic tasks, AgenticEvolve improves final quality with all three models on hidden and unseen instances and reaches baseline quality with fewer formal evaluations; on twelve equation problems it recovers more equations than the compared baselines.
Source: §VIII · p. 11Takeaways & Limitations
Experiments guide candidate construction using self-generated instances for heuristic optimization and available data for equation discovery.
Source: §VIII · p. 11Each equation-discovery method was run once, all three models were Flash-tier, and coverage was limited to two heuristic tasks and twelve benchmark equations.
Source: §VII · p. 10
Abstract
from arXiv · showhide
Evolutionary program search with large language models keeps the structure of an evolutionary algorithm and uses the model as its variation operator. Typically, the model generates each candidate program in a single request from parent programs and their scores. We present AgenticEvolve, an evolutionary program search framework in which each island is run by a persistent large language model agent. Between evaluations, the agent designs and runs experiments on the candidate program. To measure the contribution of experimentation, we compare AgenticEvolve with No-analysis, an otherwise identical configuration with experiments disabled. On two heuristic design tasks with three large language models, AgenticEvolve produces better programs than No-analysis and OpenEvolve on both hidden and unseen instances. It reaches the baselines' final quality with far fewer evaluations and comparable cost. On twelve equation discovery problems, it recovers the ground-truth structure of more equations than No-analysis and existing language-model and genetic programming baselines.
I. INTRODUCTION
Earlier LLM-based evolutionary search learns about program behavior mainly through formal evaluations, which consume budget and may use hidden instances. AgenticEvolve gives persistent agents tools to run self-designed experiments between evaluations, then tests their contribution against a matched configuration without experiments.
A. Evolutionary Program Search
- In conventional LLM evolutionary search, candidates are generated from parent programs and scores, while empirical information about behavior enters only through formal evaluation.Those evaluations consume budget and often use instances hidden from the model.Source: §I · p. 1
I. INTRODUCTION
- AgenticEvolve uses persistent LLM agents on evolutionary-search islands to run self-designed experiments between formal evaluations, with No-analysis disabling experiments while leaving other search mechanisms fixed.The paper evaluates the contribution of experimentation across heuristic design and scientific equation discovery.Source: §I · pp. 1–2
- On two heuristic tasks, AgenticEvolve has the best mean final quality with every tested model on hidden and unseen instances, reaches baseline final quality with fewer evaluations, and has model- and budget-dependent cost advantages.For equation discovery, it recovers more ground-truth equations than No-analysis and existing language-model and genetic-programming baselines at cost comparable to SR-Scientist.Source: §I · p. 2
A. Evolutionary Program Search
- AgenticEvolve retains population-based search, archives, and islands but lets a persistent agent develop candidates through experiments between formal evaluations.The matched No-analysis configuration disables this experimental capability while retaining the other search settings.Source: A. Evolutionary Program Search · p. 2, B. Agentic Search · p. 2
B. Agentic Search
- AgenticEvolve combines persistent agents with population-based search, MAP-Elites archives, and periodic migration, unlike approaches that remove the external search controller.The paper measures experiment contributions by disabling the experiment tool while retaining other search settings.Source: B. Agentic Search · p. 2
C. Scientific Equation Discovery
AgenticEvolve combines persistent island agents and evolutionary archives with experiments between formal evaluations. For equation discovery, agents can analyze available training data; across the framework, agents retain a conversation history, submit candidates for evaluation, and can access archive and migration candidates.
C. Scientific Equation Discovery
- Scientific equation discovery is studied alongside genetic-programming, LLM-based program-evolution, and agentic-search approaches, including comparisons with PySR, LLM-SR, and SR-Scientist.The cited SR-Scientist ablation reports that removing data analysis lowers the fraction of problems fitted within a 0.01 tolerance from 63.6% to 35.7% for one model, with smaller reductions for two others.Source: C. Scientific Equation Discovery · p. 2
III. AGENTICEVOLVE
- Each island is run by a persistent LLM agent that uses tools to inspect and modify a candidate, run experiments, and commit it for formal evaluation.The framework returns evaluation results with an archive summary that periodically includes neighboring islands’ elite programs.Source: §III · p. 2, Figure 1 · p. 3
A. Problem and Task Interface
- A task supplies a natural-language description, initial program, and evaluator; the returned program is the highest-fitness successfully evaluated commit across islands.Every island starts from the same initial program, and tasks may provide data for experiments.Source: A. Problem and Task Interface · p. 3
B. One Island: Persistent Conversation and Experiments
- The persistent conversation records program revisions, experiments, and evaluation results, while the agent writes and executes analysis.py between commits and receives its printed output.The experiment script can import the current program and task-provided data, but the evaluator and its hidden inputs are unavailable in its working directory.Source: B. One Island: Persistent Conversation… · p. 3
C. Archive and Islands
- The archive stores commits and organizes successful programs in a MAP-Elites grid, while ring-arranged islands periodically share elite migration candidates for the agent to consider.The agent decides whether to retrieve a migration candidate and use it to revise or replace its current program.Source: C. Archive and Islands · pp. 3–4
D. Evaluation Protocol and the No-analysis Configuration
The evaluation compares AgenticEvolve with No-analysis, which removes experiment tools while retaining the other settings, across hidden-instance heuristic tasks and equation discovery. The heuristic study uses aligned methods and models, fixed evaluation budgets, and repeated runs with specified quality and cost measures.
D. Evaluation Protocol and the No-analysis Configuration
- No-analysis removes run_analysis, access to analysis.py, and related prompt instructions while retaining the persistent conversation, editing, evaluation, archive, migration, compaction, and budget.The configuration difference is designed to measure the contribution of agent experiments.Source: D. Evaluation Protocol and the… · p. 4
IV. EXPERIMENTAL DESIGN
- For OBP and DJSS, formal evaluation instances are hidden, so agent experiments cannot reproduce the formal evaluation; equation discovery instead provides training data for direct analysis.The heuristic tasks are online bin packing and dynamic job shop scheduling.Source: §IV · p. 4
A. Tasks
- DJSS evaluates dispatching rules on five utilization-and-allowance scenarios with hidden seeds, then tests the same scenarios with new seeds rather than new scenarios.Each simulation warms up on 500 jobs and records mean tardiness for the following 2,000 jobs, normalized to rule R1.Source: A. Tasks · p. 4
B. Methods and Models
- AgenticEvolve and OpenEvolve are aligned on islands, MAP-Elites features, task, initial program, evaluator, execution limit, model, reasoning effort, and output limit; OpenEvolve generates each candidate in a single request.AgenticEvolve may run up to ten analyses per commit, and the study uses three Flash-tier models.Source: B. Methods and Models · p. 4
C. Budgets, Resource Measures and Statistics
- Each AgenticEvolve and No-analysis run evaluates 90 candidates beyond the initial program, while OpenEvolve runs up to 180 candidate evaluations, so comparisons are also made at common budgets.Formal-evaluation counts include initial, failed, and timed-out evaluations; estimated LLM cost uses recorded token usage and model prices.Source: C. Budgets, Resource Measures and… · pp. 4–5
- Each method is run five times per model, and final-quality differences use unpaired two-sided Mann–Whitney U tests at the 0.05 level with bootstrap 95% confidence intervals for mean differences.Tasks, budgets, and stopping conditions were fixed before formal runs; heuristic runs predated the later feedback restriction, and an audit found no access to evaluators or hidden instances.Source: C. Budgets, Resource Measures and… · p. 5
V. HEURISTIC OPTIMIZATION
AgenticEvolve improves final heuristic quality on hidden and unseen instances, and often reaches OpenEvolve’s final quality with fewer formal evaluations. Its quality-versus-cost advantage varies by task and model.
A. Final Quality and Generalization
- On unseen instances, AgenticEvolve has the lowest mean on every set with every model; OR1 excess is 2.9–4.1% versus 5.3% for FunSearch and 5.8% for Best Fit, while fresh-seed DJSS tardiness is 0.84–0.86 versus 0.92–1.00 for the other methods.Fresh OBP sequences from the training distribution show the same pattern.Source: A. Final Quality and Generalization · p. 5
B. Quality Against Formal Evaluations
- In five of six model-task cases, AgenticEvolve reaches OpenEvolve’s final quality within 20 formal evaluations, compared with OpenEvolve’s 180-iteration runs.Its best-so-far curve separates within the first ten evaluations and stays ahead through its budget.Source: B. Quality Against Formal Evaluations · p. 5
C. Quality Against Cost
- AgenticEvolve’s mean LLM cost is at most 4.5 USD per run, but it costs more than No-analysis and costs less than OpenEvolve only with DeepSeek.OpenEvolve’s long DeepSeek outputs dominate its cost, whereas repeated cached input accounts for approximately 78% and 87% of AgenticEvolve’s cost with Qwen and GLM, respectively.Source: C. Quality Against Cost · pp. 5–6
- At equal cost, AgenticEvolve outperforms OpenEvolve with DeepSeek and Qwen, reaching its final quality for 0.08–0.19 USD versus 0.64–3.06 USD for a complete OpenEvolve run; GLM quality is similar over the covered cost range.Against No-analysis, AgenticEvolve is better on OBP across models, while on DJSS No-analysis improves cheaply at first and AgenticEvolve catches up near or above its completed-run cost.Source: C. Quality Against Cost · p. 5
A. Final Quality and Generalization
- On hidden instances, AgenticEvolve has the best mean quality with every model on both tasks, reducing OBP excess over the L2 bound by 1.8–2.3 percentage points versus No-analysis and 1.1–2.4 versus OpenEvolve, and DJSS normalized tardiness by 0.08–0.16 and 0.07–0.16, respectively.The confidence intervals of all twelve differences exclude zero; nine are significant, while no No-analysis versus OpenEvolve difference is significant.Source: A. Final Quality and Generalization · p. 5, Table II · p. 6
D. A Search Trajectory
A traced OBP commit shows how the agent used self-generated sequences to diagnose waste and optimize a new placement rule. The symbolic-regression protocol evaluates twelve selected problems using a shared evaluator and common structure-recovery criteria.
D. A Search Trajectory
- Bins with less than 20 units of remaining capacity accounted for 58% of Best Fit’s total waste in the agent’s 400 generated sequences.The agent used these sequences to estimate final waste U(L); above 20, the estimate fell to 7–11 units and stayed in that range at larger observed capacities.Source: D. A Search Trajectory · p. 7
- The new OBP rule reduced hidden-set excess from 6.553% to 5.392% after the agent optimized an estimated-waste potential from 400 self-generated sequences.It scored feasible placements by the change in estimated final waste, with a small Best Fit tie-breaker; self-generated-sequence excess fell from 6.3–6.6% under Best Fit to 5.4–5.7%.Source: D. A Search Trajectory · p. 7
VI. SYMBOLIC REGRESSION
- Symbolic regression compares AgenticEvolve and No-analysis with LLM-SR, SR-Scientist, and PySR on twelve selected LLM-SRBench problems.The in-domain training data are available during analysis, while in-domain and out-of-domain test sets are held back for reporting.Source: §VI · p. 7
A. Protocol
- The protocol uses a common least-squares evaluator for LLM-based methods and assesses exact or negligible-extra-term recovery, with material-science recovery reduced along the coupled-input trajectory.The twelve problems meet prespecified selection criteria; LLM-based runs use up to 1,000 model requests per problem, with up to ten analyses per commit for AgenticEvolve.Source: A. Protocol · pp. 7–8
B. Accuracy and Structure Recovery
AgenticEvolve recovers structure on more chemistry, biology, and physics problems than the comparison methods, while material-science recovery is constrained by coupled inputs. Its resource use and a BPG18 case study illustrate how analysis can guide search and fit a difficult constant.
B. Accuracy and Structure Recovery
- AgenticEvolve meets the out-of-domain tolerance on all twelve problems, compared with eight for No-analysis and SR-Scientist, three for LLM-SR, and one for PySR.Meeting the predictive-accuracy tolerance does not always mean structural recovery: the passage identifies differing structures for No-analysis on PO1, SR-Scientist on CRK21, and several methods on MatSci11.Source: B. Accuracy and Structure Recovery · p. 8
- AgenticEvolve recovers structure on all nine chemistry, biology, and physics problems, versus four each for No-analysis and SR-Scientist and one each for LLM-SR and PySR.AgenticEvolve recovers six exactly and three with negligible extra terms; on material science, the coupled-data trajectory leaves independent temperature dependence undetermined, and no method recovers MatSci11’s reduced form at the stated tolerance.Source: B. Accuracy and Structure Recovery · pp. 8–9
- AgenticEvolve’s candidate sets contain exact equations for eleven problems, although final selections retain negligible extra terms for five of them.No-analysis and SR-Scientist each find exact equations for five problems, while LLM-SR and PySR find one each.Source: B. Accuracy and Structure Recovery · p. 9
C. Search Resources
- At comparable total cost, AgenticEvolve recovers eleven equations exactly or with negligible extra terms, versus six for SR-Scientist, at costs of 6.5 and 6.6 USD, respectively.AgenticEvolve made 316 commits in 4,939 requests, compared with 3,988 commits in 7,809 requests for No-analysis; it experimented locally before committing.Source: C. Search Resources · p. 9
D. Analysis Case
- For BPG18, every method submitting a free-frequency model obtained NMSE between 0.93 and 0.97, while AgenticEvolve islands scanned frequencies and recovered exact structure.Two islands did so on their first commit after ten analyses; the third recovered it on its second commit after four further analyses.Source: D. Analysis Case · p. 9
- No-analysis reached the exact-recovery criterion for BPG18 at commit 107 with ω = 0.77605, while SR-Scientist submitted the exact structure at its second evaluation.LLM-SR first recovered the ground truth with negligible extra terms at its 209th sample; PySR selected a different structure with out-of-domain NMSE 3.4 × 10^-3.Source: D. Analysis Case · p. 9
VII. DISCUSSION
The discussion explains how agent-designed experiments support candidate construction while identifying limits in cost, evaluation security, and the evidence for individual search mechanisms. The conclusion reports gains across both task settings and leaves the necessary role of evolutionary structure open.
VII. DISCUSSION
- Agents used analysis to construct and locally test candidates, with the procedure depending on available data: symbolic-regression training data are inspectable, while heuristic evaluation instances are hidden.Examples include measuring Best Fit waste and scanning trial frequencies on BPG18; heuristic improvements still require verification on hidden instances.Source: §VII · pp. 9–10
- Prompt caching kept mean cost at 0.6–4.5 USD per run, but persistent conversations sent 24–164 times as many input tokens as OpenEvolve and reached compaction thresholds in 15 of 30 heuristic runs.The cost range varies across models and tasks; cached input made up different shares of cost across models.Source: §VII · p. 10
- Four of 90 final heuristic programs tuned parameters at import time, making scores sensitive to evaluator-machine load; hidden-data access restrictions were protocol-level, with enforcement requiring an operating-system sandbox.Re-evaluation changed one such program’s score by less than 10^-3 and left the other three unchanged; an audit found no successful hidden-data access.Source: §VII · p. 10
- Cross-island reads preceded improvements in many commits, but their association does not establish migration’s independent effect.The authors note that measuring that effect would require comparison with migration disabled.Source: §VII · p. 10
- Evidence is bounded by twelve equation problems, two heuristic tasks, one symbolic-regression run per problem, and three Flash-tier models.Whether stronger models help, or No-analysis could close the quality gap with more evaluations at AgenticEvolve’s total cost, remains untested.Source: §VII · p. 10
VIII. CONCLUSION
- AgenticEvolve improved final quality on two heuristic tasks with all three models, transferred gains to unseen instances, and recovered more equations than four baselines.It reached No-analysis and OpenEvolve quality with a fraction of their formal evaluations; Table IV reports selected equation structure and out-of-domain normalized mean squared error, judging material-science problems against reduced truth.Source: Table IV · p. 10, §VIII · p. 11
- How much population, archive, island, and migration structure an agent-driven search needs remains open.The paper adopted these mechanisms from evolutionary computation, where they were developed for predefined variation operators.Source: §VIII · p. 11
A. Model Requests
The implementation specifies model-request handling, baseline configurations, and the tools and workflow given to persistent agents. It also distinguishes OpenEvolve’s setup from AgenticEvolve and details the symbolic-regression comparison systems.
A. Model Requests
- AgenticEvolve streams requests with tool schemas and a 300-second inter-chunk timeout, while OpenEvolve uses non-streaming requests and a 1200-second whole-request timeout.AgenticEvolve limits context to 500,000 tokens and compacts conversations above 70% of that limit; compaction occurred in 15 of 30 heuristic runs.Source: A. Model Requests · p. 13
B. OpenEvolve Settings
- OpenEvolve retained defaults except fixed settings, and differed in feature normalization, invalid-diff evaluation counting, and metric rounding.OpenEvolve rescales features using observed minima and maxima, while AgenticEvolve uses fixed evaluator normalization; OpenEvolve displays metrics to four decimals.Source: B. OpenEvolve Settings · p. 13, Table S1 · p. 13
C. Symbolic-Regression Baselines
- The symbolic-regression comparisons used the LLM-SR and SR-Scientist searches with shared evaluator rules, alongside PySR’s published benchmark settings.The implementations and settings differ: LLM-SR uses ten islands, SR-Scientist runs up to 40 rounds, and PySR uses 125 iterations of 550 cycles.Source: C. Symbolic-Regression Baselines · p. 14
D. System Prompt
- Each island’s agent can investigate hypotheses with analysis scripts, edit the candidate program, and submit commits for formal evaluation and feedback.The system prompt directs agents to form a hypothesis, modify the program, commit and inspect feedback, then choose another idea; No-analysis removes the analysis tool and its instructions.Source: D. System Prompt · pp. 14–15
E. Framework Versions
The supplementary sections document framework versions, final-quality checks, budget measurement, and an experimental search trajectory. They also report a representative bin-packing program and explain how its parameters were obtained.
E. Framework Versions
- Heuristic runs used framework versions 0.1.0–0.1.2, while symbolic-regression runs used version 0.2.0 for four methods and 0.2.1 for No-analysis.The heuristic version changes affected request attempts and generalization scripts, not search, evaluators, or budgets.Source: E. Framework Versions · p. 15
A. Final Quality and Generalization
- Re-evaluated hidden-set fitness matched recorded values to six decimal places for every final program except one DeepSeek No-analysis OBP program, whose fitness changed by less than 10^-3.The exception tuned parameters at import time within a wall-clock limit.Source: A. Final Quality and Generalization · p. 15
B. Budgets to Reach OpenEvolve
- The budget analysis records the first evaluation and cost budgets at which AgenticEvolve’s mean best-so-far quality reaches OpenEvolve’s mean final quality; the two budgets need not describe the same search point.Completed runs retain their final quality at larger cost budgets.Source: B. Budgets to Reach OpenEvolve · p. 16, Table S4 · p. 17
C. A Search Trajectory
- The committed bin-packing program achieved 5.392% hidden-set excess versus 6.553% for Best Fit; its knot values came from coordinate descent, not the commit description’s policy-iteration account.The examined commit followed ten analyses on sequences of 120 items with sizes 20–100 and capacity 150.Source: C. A Search Trajectory · p. 17, Table S5 · p. 17
D. Cross-Island Reads
The supplementary sections quantify cross-island reads, specify equation-recovery criteria, and show selected equations. For material-science tasks, assessment uses reduced truths that do not retain all original temperature dependence.
D. Cross-Island Reads
- Of 215 commits following cross-island reads, 36 contained code identical to a read program and one differed only in whitespace; the logs also distinguish offered candidates from those actually read.There were 25 reads not represented in the island’s recorded migration feedback; these contribute only to commit-based statistics.Source: D. Cross-Island Reads · p. 18, Table S6 · p. 18
A. Structure Assessment
- An equation counts as recovered only with the required structure and training error below 10^-3 of target standard deviation; missing ground-truth terms cannot be treated as negligible extras.For MatSci6 and MatSci9, the reduced truths omit original temperature dependence, and no method recovered MatSci11’s retained Arrhenius factor.Source: A. Structure Assessment · pp. 18–19
B. Selected Equations
- For PO26, the selected AgenticEvolve equation is marked exact at 22 nodes, while the 28-node No-analysis equation is marked not recovered.The table defines exact and not-recovered classifications and judges material-science equations against reduced truths.Source: Table S7 · p. 19, B. Selected Equations · p. 20
C. Effect of the Final Selection
In seven runs, an exact equation appeared among the candidates, but the selected equation had negligible extra terms.
C. Effect of the Final Selection
- In seven runs, exact candidates existed but selection returned equations with negligible extra terms; five AgenticEvolve cases had the exact candidate as the smallest on the Pareto front.In the five AgenticEvolve cases, the exact candidate was eligible, but a larger candidate with a slightly lower training error received a positive selection score; in two other runs, exact candidates were excluded or scored lower.Source: C. Effect of the Final Selection · p. 21
D. Accuracy and Resources
The supplementary material reports selected-equation in-domain and out-of-domain errors, along with each method’s token composition, cost, and wall time.
D. Accuracy and Resources
- Tables S8 and S9 report selected-equation errors and method resource measures, respectively.The listed resource measures are token composition, cost, and wall time.Source: D. Accuracy and Resources · p. 21
S4. EVALUATION-ACCESS AUDITS
The audits examine whether search runs accessed hidden evaluation data, concluding that final runs obtained no data beyond their methods’ inputs while noting limits in the audit record.
S4. EVALUATION-ACCESS AUDITS
- The audit reviewed recorded code, evaluation feedback, analysis output, and model reasoning without executing recorded programs; candidate programs and scripts ran without an operating-system sandbox.Access restrictions formed part of the protocol, and compliance was checked after the runs.Source: S4. EVALUATION-ACCESS AUDITS · p. 21
A. Heuristic Design Runs
- 120 of 131 failed heuristic evaluations exposed the evaluator’s absolute path, yet no heuristic run used it, read outside its temporary directory, or reconstructed hidden instances.Two No-analysis runs on OBP returned computed values through exception messages.Source: A. Heuristic Design Runs · p. 22
B. Symbolic Regression Runs
- LLM-SR samples used no file or process operations, introspection, or value output; SR-Scientist read training data from standard input as instructed, with neither method’s code accessing files or host paths.The symbolic-input evaluator fitted expressions itself, so candidates did not receive data values.Source: B. Symbolic Regression Runs · pp. 22–23
- Earlier No-analysis BPG18 runs loaded training data and returned scan results in exceptions, prompting the feedback restriction and candidate-load check in version 0.2.1.In the final symbolic-regression runs, 14 commits attempted to obtain evaluation values without success; AgenticEvolve analyses also unsuccessfully searched for test data or evaluator code.Source: B. Symbolic Regression Runs · p. 23
- Final runs obtained no data beyond each method’s inputs, but the audit cannot exclude effects absent from its records.For LLM-SR, records contained extracted function bodies rather than full model responses.Source: B. Symbolic Regression Runs · p. 23