Source-linked AI summary
Code Evolution for Control: Synthesizing Policies via LLM-Driven Evolutionary Search
Ping Guo, Chao Li, Yinglan Feng, Chaoning Zhang
TL;DR
RL can require high sample complexity and produce opaque policies, while manual design requires substantial expertise and scales poorly across tasks. The paper treats policy synthesis as code evolution, using LLMs to generate and recombine code variations within evolutionary search. On the LunarLander benchmark, EvoEngineer achieved 7× higher reward than FunSearch and EoH at the same LLM budget; EvoEngineer+ reached a 70% success rate versus PPO’s 60%, despite lower average reward.
Problem
RL can require high sample complexity and produce opaque policies, while manual design requires substantial expertise and scales poorly across tasks.
Source: §I · p. 1Method
The paper treats policy synthesis as code evolution, using LLMs to generate and recombine code variations within evolutionary search.
Source: §I · p. 1Results
On the LunarLander benchmark, EvoEngineer achieved 7× higher reward than FunSearch and EoH at the same LLM budget; EvoEngineer+ reached a 70% success rate versus PPO’s 60%, despite lower average reward.
Source: A. Main Results · p. 3, §VII · p. 5Takeaways & Limitations
The authors identify structured behavioral feedback, rather than fitness scores alone, as key to synthesizing effective control logic in their study.
Source: §VII · p. 5Evaluation covers only LunarLander-v3 with discrete actions; generalization to continuous control, higher-dimensional states, and more complex tasks remains un demonstrated.
Source: D. Limitations · p. 5
Abstract
from arXiv · showhide
Designing effective control policies for autonomous systems remains a fundamental challenge, traditionally addressed through reinforcement learning or manual engineering. While reinforcement learning has achieved remarkable success, it often suffers from high sample complexity, reward shaping difficulties, and produces opaque neural network policies that are hard to interpret or verify. Manual design, on the other hand, requires substantial domain expertise and struggles to scale across diverse tasks. In this work, we demonstrate that LLM-driven evolutionary search can effectively synthesize interpretable control policies in the form of executable code. By treating policy synthesis as a code evolution problem, we harness the LLM's prior knowledge of programming patterns and control heuristics while employing evolutionary search to explore the solution space systematically. We implement our approach using EvoToolkit, a framework that seamlessly integrates LLM-driven evolution with customizable fitness evaluation. Our method iteratively evolves populations of candidate policy programs, evaluating them against task-specific objectives and selecting superior individuals for reproduction. This process yields compact, human-readable control policies that can be directly inspected, modified, and formally verified. This work highlights the potential of combining foundation models with evolutionary computation for synthesizing trustworthy control policies in autonomous systems. Code is available at https://github.com/pgg3/EvoControl.
I. INTRODUCTION
Effective control policies are difficult to design: reinforcement learning can be sample-intensive and opaque, while manual design requires expertise and scales poorly. The paper treats policy synthesis as code evolution, applying LLM-driven evolutionary search to generate interpretable policies and compare evolution strategies on LunarLander.
I. INTRODUCTION
- RL policies can require high sample complexity and difficult reward shaping, while manual policy design demands expertise and scales poorly across tasks.Neural-network policies can also resist interpretation and verification.Source: §I · p. 1
- Programmatic policies are transparent and verifiable, but manual implementation is expertise-intensive and automated search faces DSL overhead or combinatorial explosion.The paper identifies these as barriers to synthesizing effective programmatic policies.Source: §I · p. 1
- Prior LLM-driven code evolution has been used in other domains, but its application to control policy synthesis remains unexplored.Examples include mathematical construction, symbolic regression, heuristic optimization, and CUDA kernels.Source: §I · p. 1
- The proposed approach uses LLMs as mutation and crossover operators, combining meaningful code variations with evolutionary search over the solution space.It draws on programming patterns and control heuristics.Source: §I · pp. 1–2
- The study compares FunSearch, EoH, and EvoEngineer on LunarLander and reports competitive performance with better interpretability than neural-network alternatives.These are the stated contributions and result claims in the introduction.Source: §I · p. 2
A. Reinforcement Learning for Control
Prior work describes limitations of reinforcement learning and program synthesis, alongside LLM-driven code evolution across several domains. This paper extends that line to evolve interpretable Python policies mapping states to actions.
A. Reinforcement Learning for Control
- RL methods can require millions of environment interactions and produce opaque neural-network policies.The passage cites DQN and PPO [1] as examples of value-based and policy-gradient methods.Source: A. Reinforcement Learning for Control · p. 2
B. Program Synthesis and Neuro-Symbolic Methods
- Program synthesis and neuro-symbolic approaches can improve interpretability, but often require domain-specific languages or extensive supervision.Source: B. Program Synthesis and… · p. 2
C. LLM-Driven Code Evolution
- LLM-driven evolution has been applied to mathematical construction, symbolic regression, heuristic optimization, and CUDA kernels; this work extends it to interpretable Python policies mapping states to actions.The cited examples are FunSearch, CoEvo, EoH, and EvoEngineer [9].Source: C. LLM-Driven Code Evolution · p. 2
III. METHODOLOGY
The paper formulates control policy synthesis as finding a policy that maximizes expected cumulative reward in an environment. It constrains the policy to interpretable Python code that can be inspected, modified, and verified.
A. Problem Formulation
- The objective is to find a policy mapping states to actions that maximizes expected cumulative reward in an environment.The environment is characterized by its state space, action space, and reward function.Source: A. Problem Formulation · p. 2
- The policy is constrained to an interpretable Python function that can be inspected, modified, and verified.Source: A. Problem Formulation · p. 2
B. LLM-Driven Evolution Framework
The framework evolves a population of candidate policy programs using LLM-based operators, with three strategies differing in how they prompt or structure evolution. Fitness is based on average reward across episodes, and execution is sandboxed.
B. LLM-Driven Evolution Framework
- The framework maintains a population of candidate policies and evolves them using LLM-based operators.The procedure is summarized in Algorithm 1.Source: B. LLM-Driven Evolution Framework · p. 2
- FunSearch uses a single prompt, EoH uses five specialized operators, and EvoEngineer supplies episode statistics, failure modes, and domain hints.EvoEngineer also requests structured responses containing code and rationale.Source: B. LLM-Driven Evolution Framework · p. 2
C. Fitness Evaluation
- For control tasks, fitness is the average reward over multiple evaluation episodes.Policies run in a sandbox with restricted builtins; invalid actions or runtime errors receive minimum fitness.Source: C. Fitness Evaluation · pp. 2–3
IV. EXPERIMENTS
The experiments compare three LLM-driven evolution strategies with random and PPO baselines on LunarLander-v3. At a comparable environment-step budget, EvoEngineer+ has higher success than PPO, while PPO has higher average reward.
A. Experimental Setup
- LunarLander-v3 provides an 8-dimensional continuous state, four discrete actions, and rewards for landing, crashes, and fuel use; success is cumulative reward ≥200.The three LLM methods use GPT-4o, a population of 10, and 20 generations; fitness averages reward over 10 episodes, with results averaged across five runs.Source: A. Experimental Setup · p. 3
A. Main Results
- With an identical budget of approximately 45 LLM calls, EvoEngineer achieves average reward 66.6 and 40% success, outperforming FunSearch and EoH, whose rewards remain near zero (9.1 and 6.2).The passage describes this as a 7× improvement in average reward over the simpler approaches.Source: A. Main Results · p. 3
- At approximately 1M environment steps, EvoEngineer+ reaches 70% success versus PPO’s 60%, while PPO achieves higher average reward.EvoEngineer+ uses 200 LLM calls; the comparison reflects a trade-off between landing success and average reward.Source: Table II · p. 3, A. Main Results · p. 3
B. Convergence Analysis
EvoEngineer converges faster than EoH in the reported runs, though outcomes vary across seeds. At comparable environment-interaction budgets, EvoEngineer+ has higher success than PPO, but PPO has higher average reward.
B. Convergence Analysis
- EvoEngineer reaches positive rewards within the first few generations and consistently outperforms EoH, though both continue improving and results vary across seeds.The figure plots best fitness against generation; shaded regions show standard deviation across five seeds.Source: B. Convergence Analysis · p. 3, Figure 1 · p. 4
C. Sample Efficiency Analysis
- At equivalent sample budgets, EvoEngineer+ achieves a 70% success rate versus PPO’s 60%, while PPO’s average reward is higher (214 vs 144).The comparison uses approximately 1M environment steps; code evolution’s discrete policies may excel in some episodes and fail in others, unlike PPO’s expected-return optimization.Source: C. Sample Efficiency Analysis · pp. 3–4
- The environment-step comparison does not capture the methods’ different costs: EvoEngineer+ incurs non-trivial LLM inference expense, whereas PPO’s training uses local GPU computation.The passage notes that this distinction may diminish as LLM inference costs fall and reverses in real-world robotics, where physical interactions are expensive and potentially dangerous.Source: C. Sample Efficiency Analysis · p. 4
D. Evolved Policy Analysis
The best evolved policy uses altitude-based phases and readable state thresholds to map observations to actions. The discussion attributes EvoEngineer’s stronger results over simpler evolution strategies to feedback that gives the LLM context about episodes and failures.
D. Evolved Policy Analysis
- The best EvoEngineer+ policy uses three altitude-based control phases, with human-readable thresholds and direct mappings from state observations to discrete actions.The full policy also handles angular velocity and leg-contact states beyond the simplified excerpt.Source: D. Evolved Policy Analysis · p. 4, Figure 2 · p. 4
A. The Role of Context in LLM-Driven Evolution
- Without explicit feedback, the LLM can generate syntactically valid code but lacks signals to diagnose policy failures or how to improve them.EvoEngineer feeds episode statistics, failure-mode analysis, and domain hints back to the model; the passage gives high descent velocity at low altitude as a crash-related cue for earlier braking.Source: A. The Role of Context in LLM-Driven… · p. 4
- EvoEngineer’s average reward of 66.6 versus approximately 8 for simpler approaches is presented as evidence that context, not just evolutionary structure, drives effective policy synthesis.The feedback loop is described as transforming the LLM from a blind code mutator into an informed policy designer.Source: A. The Role of Context in LLM-Driven… · p. 4
B. When to Choose Evolved Code over Neural Networks
EvoEngineer+ trades lower average reward for higher success than PPO, a distinction that matters depending on whether task completion or consistency is the priority. With more LLM calls, the evolved policy improves while becoming shorter; its API-centered cost profile may suit settings where physical interactions are expensive or dangerous.
B. When to Choose Evolved Code over Neural Networks
- EvoEngineer+ achieves higher success than PPO (70% vs 60%), while PPO earns higher average reward (214 vs 144).Success rate may better fit binary task outcomes, whereas average reward captures consistency across episodes.Source: B. When to Choose Evolved Code over… · p. 4
- A 59-line Python function can be reviewed, audited, and certified, with decision boundaries that domain experts can inspect and modify without retraining.The passage contrasts this with a 10K-parameter neural network.Source: B. When to Choose Evolved Code over… · p. 5
C. Scaling and Practical Considerations
- Scaling from 45 to 200 LLM calls raises reward from 66.6 to 143.6 and success from 40% to 70%, while reducing policy length from 80 to 59 lines.The passage suggests later iterations refine edge cases and remove redundant logic.Source: C. Scaling and Practical Considerations · p. 5
- LLM API calls dominate code evolution’s budget, while physical environment interactions can be costly or dangerous in real-world robotics.The passage contrasts this with PPO, for which GPU computation is the primary cost and simulation steps are cheap.Source: C. Scaling and Practical Considerations · p. 5
D. Limitations
The study reports that rich contextual feedback improves evolved-policy performance, while its evaluation and practical applicability remain bounded by benchmark scope, model dependence, performance variance, and API costs. Future work would extend evaluation and explore hybrid and multi-objective approaches.
D. Limitations
- Evaluation is limited to discrete-action LunarLander-v3; generalization to continuous control, higher-dimensional states, and more complex tasks remains un demonstrated.The approach also inherits LLM biases and capabilities, so performance may vary across models and degrade outside their training distributions.Source: D. Limitations · p. 5
- Evolved policies show higher variance than neural networks, and extended evolution of 200+ calls still incurs non-trivial API costs compared with local GPU training.Policies may succeed decisively or fail completely rather than deliver consistent moderate performance.Source: D. Limitations · p. 5
VII. CONCLUSION
- EvoEngineer achieved 7× higher reward than FunSearch and EoH with the same LLM budget; EvoEngineer+ reached 70% success versus PPO’s 60%, despite lower average reward.EvoEngineer+ produced interpretable 59-line policies.Source: §VII · p. 5
- Future work includes continuous action spaces, complex locomotion and manipulation tasks, hybrid LLM evolution with gradient-based fine-tuning, and multi-objective optimization of performance, interpretability, and robustness.Source: §VII · p. 5