Evolution through Large Models

1 Introduction

• ELM uses LLM-suggested, intelligent mutations to make program evolution more effective, and can fine-tune its mutation model on successful changes for iterative self-improvement.The authors frame this as a way for LLMs and evolution to guide one another toward solutions neither could reach alone with conventional mutation or standalone generation.Source: §1 · pp. 2–3
• ELM links genetic programming and LLMs by using models trained on code to approximate intentional human program changes while preserving functionality.The paper presents this synergy as a way to evolve programs in domains far outside an LLM’s training distribution, where the model alone has limited recourse and conventional GP may struggle to obtain a consistent fitness signal.Source: §1 · p. 2
• The experiments generate a Sodarace dataset from one mediocre human-designed program, encoding robots as raw Python programs for a generic code-generating LLM.Sodarace features two-dimensional ambulating robots with arbitrary morphologies for diverse terrains [11, 12].Source: §1 · pp. 2–3
• Generating diverse solutions where training data is scarce may support open-ended search beyond prior experience, and ELM-generated data offers a route to that capability.The paper focuses on this data-generation implication while noting that ELM may apply in other contexts.Source: §1 · pp. 2–3
• ELM-generated data can fine-tune LLMs to consistently produce viable Sodaracers, while reinforcement learning enables an augmented LLM to generate them conditionally on terrain.The introduction presents this conditional invention as a potential foundation for new open-ended processes.Source: §1 · p. 3

2.1 Genetic Programming

• GP could evolve broadly capable programs, but scaling and domain-specific setup constrain it, while random code mutations remain unlike humans’ deliberate, coupled edits.Researchers often must specify available functions, variables, and control structures, limiting what evolution can produce.Source: §2.1 · p. 4

2.2 Large Language Models

• GitHub-trained code models synthesize functions [9] [10], while diff models use code and optional commit intent for informed, controllable mutations that may adapt through gradient descent.General language models can also act like diff models when given suitable prompts.Source: §2.2 · p. 5

2.3 Open-endedness

• Open-endedness seeks never-ending innovation, yet environments limit what can arise and remain interesting; the paper proposes inventions responding to prior inventions as a route to continuing open-endedness.Prior approaches have explored more complex environments, including video games [38], or co-evolving environmental features with agents [40] [42].Source: §2.3 · p. 5
• Existing encodings often describe fixed environment components [40], whereas computer programs offer a general encoding for continually expanding environmental richness.The paper identifies a lack of theory about which forms of added environmental complexity support enduring open-endedness.Source: §2.3 · pp. 5–6

3.1 Mutation through Diff

• A diff model predicts code changes from a file and commit message, learning plausible edits from human-written diffs rather than merely generating or memorizing the input code.ELM uses natural-language commit messages to specify the mutation; their wording can calibrate its scope, and other messages are possible.Source: §3.1 · pp. 6–7
• On 4-Parity, diff mutation fixed all five introduced bugs, while GP mutation produced no solution in 100,000 trials and became exponentially less successful as bugs accumulated.Diff mutation was affected more by the fifth bug, which changed the modulo calculation, than by the number of bugs alone.Source: §3.1 · p. 7, Figure 1 · p. 8
• On 4-Parity, API edit-mode models and few-shot prompting outperformed the paper’s diff model as alternative LLM-based mutation approaches.The prompting approach can use a code-trained model; the passages describe both alternatives as viable options for building on the paper’s work.Source: §3.1 · p. 7, Figure 2 · p. 9

3.2 The Evolutionary Algorithm and Implications for Open-Endedness

• ELM can use modern programming languages, including variable-length Python genotypes, without special accommodation, and its modular mutation operator can fit diverse evolutionary algorithms.The utility of natural-language or other encodings depends on how they map to a potentially useful phenotype.Source: §3.2 · pp. 7–8
• The section frames ELM with quality-diversity search as an opportunity to address open-ended algorithms’ tendency to plateau, while noting that tractable strong open-endedness remains unresolved.It connects the challenge to finite artificial environments and proposes that agents may need to alter their environment through artifacts in response to existing ones.Source: §3.2 · pp. 8–10
• The paper uses MAP-Elites, which mutates solutions sampled from occupied niches and retains candidates that fill a new niche or outperform its champion, building a diverse map of high-quality programs.Quality-diversity search supplies varied artifacts as training data that could bootstrap an LLM into a generator of conditional artifacts, or detached conditional things.Source: §3.2 · pp. 8, 10

3.3 Fine-tuning the Diff Operator

• Fine-tuning the diff model on MAP-Elites-accepted diffs can improve mutations toward changes yielding higher quality or novelty, and potentially track changing opportunities.The paper demonstrates one fine-tuning iteration; continual refinement remains open research, and prompt-based mutation could also be fine-tuned.Source: §3.3 · pp. 11–12

4 Experiment and Results

• Sodarace is a physics-based domain in which robots made of point masses and oscillating springs are evaluated by how far they traverse a terrain.The experiment uses Python programs to describe robot morphologies, with morphology features recorded for MAP-Elites.Source: §4 · p. 12
• Without prior domain experience, an RL agent cannot bootstrap because producing even a valid artifact is effectively impossible, leaving no reward gradient.In Sodarace, the agent might observe terrain and receive reward based on the resulting robot’s performance there.Source: §4 · p. 13
• ELM combines a diff function with MAP-Elites to search the artifact space for diverse functional examples when no LLM has exposure to the target domain’s data.The examples can then train an LLM, addressing the otherwise elusive source of data needed to bootstrap the invention pipeline.Source: §4 · p. 14
• The proposed three-stage pipeline uses ELM to search for diverse example artifacts, pre-trains an LLM on them, then fine-tunes it with reinforcement learning to invent conditionally.The conditional LLM is intended to use observed conditions, such as terrain, to produce appropriate inventions.Source: §4 · pp. 12–14

4.1 Encoding Sodaracers with Python

• Python code serves as a generic invention representation: the interpreter executes each program to produce a Sodaracer data structure for simulation.The experiments use a procedural “walker creator” interface, which preliminary tests found more effective than constructing the dictionary directly; it also enforces constraints that protect the physics engine.Source: §4.1 · pp. 14–15

5 Pipeline Stage 1: Data Generation through ELM

• MAP-Elites places candidates in a 12 × 12 × 12 map by height, width, and mass, retaining each niche’s champion as diff mutations expand diversity and quality.The search starts from one evaluated hand-designed solution, then repeatedly perturbs a solution from a randomly selected occupied niche.Source: §5 · p. 17
• Four seeds explored square, radial, CPPN-fixed, and CPPN-mutable motifs; the mutable CPPN seed lets the encoding’s core functionality evolve.Source: §5 · pp. 17–18

5.1 Experimental Details and Results

• Each seed had three independent runs of 1,024,000 evaluations, using a 300M-parameter pretrained diff model for perturbations.Each run comprised 2,000 iterations of 512 diffs.Source: §5.1 · p. 18
• ELM filled a large proportion of niches across runs, but the Square seed spread more slowly because reaching high-mass niches required loops or function composition.The figure reports the percentage of 1,728 niches filled, averaged across three runs per seed.Source: §5.1 · pp. 18–19, Figure 8 · p. 19
• The CPPN seed discovered diversity fastest, while the Radial seed produced higher-quality solutions on average, showing that seed-dependent quality-diversity outcomes differ.The paper identifies the QD score as the sum of final-map champions’ performance, combining quality and the number of niches discovered.Source: §5.1 · p. 19
• Fine-tuning the diff model on accepted diffs increased performance and produced a higher percentage of valid and runnable diffs; those runs supplied later stages.Further fine-tuning showed diminishing returns in preliminary experiments.Source: §5.1 · pp. 19–20
• Although the Radial seed performed well quantitatively, its walkers appeared to exploit chaotic dynamics overfit to flat terrain, so its runs were excluded from later stages.Source: §5.1 · p. 20
• ELM combined an LLM-based mutation operator with a quality-diversity algorithm to generate hundreds of thousands of working training examples in a domain with no prior data.Source: §5.1 · p. 20

• The fine-tuning dataset includes admitted solutions above a per-niche performance threshold; the 50% and 80% cutoffs yield 280K and 95K examples, respectively.Radial-seed runs were excluded because their solutions were qualitatively chaotic and preliminary experiments suggested they harmed Stage 3 performance.Source: §6 · p. 21
• Pretrained code-generating models from 0.1M to 680M parameters were fine-tuned using standard log-probability loss, with 5% of data held out and a randomly initialized 300M control.For in-distribution quality and diversity evaluation, the selected checkpoint was the one with the lowest training loss, and 1,024 samples per model were evaluated in a new MAP-Elites map.Source: §6 · pp. 21, 23
• Fine-tuning the diff model improves the evolutionary process across the number of niches reached, QD score, and percentage of valid runnable diffs.Source: Figure 11 · p. 22
• Models above 85M parameters may not better fit the 80% threshold dataset, and random initialization worsens test loss relative to code-pretrained fine-tuning.The 50% threshold models show qualitatively similar trends, but loss values across the two datasets are not directly comparable.Source: Figure 12 · p. 23, §6 · p. 23
• LLM samples achieve performance similar to Stage 1 dataset samples, with a slight, consistent QD-score benefit from models trained on the 80% threshold dataset.The 80% dataset is less diverse but higher quality on average than the 50% dataset, and models trained on it reflect those properties.Source: §6 · p. 23, Figure 13 · p. 24
• On out-of-distribution completion tasks, 80% threshold models generally perform better, while the randomly initialized 300M model significantly underperforms.Each model produced 1,024 completions per task; performance was assessed using the checkpoint with the lowest test loss.Source: §6 · pp. 25–26, Figure 15 · p. 26

7 Pipeline Stage 3: Conditional RL

• Stage 3 turns Stage 2’s unconditional LLM conditional by mapping terrain inputs to token-embedding vectors used as a prefix, then RL fine-tunes the TEN and LLM.Episodes sample a terrain, generate a program, evaluate its Sodaracer in simulation, and reward the distance traveled.Source: §7 · pp. 26–27

7.1 Terrain Distributions

• The uniform terrain sets combine left-wall and right-wall with either tunnel or bumpy, since no single Sodaracer performs well across all terrains.Walls require distinct racers; tunnel favors short morphologies, while bumpy favors changing locomotion.Source: §7.1 · pp. 27–28, Figure 16 · p. 28

7.2 Parametrizing TENs

• Discrete terrain-token embeddings are simple and parameter-light but scale with terrain count and do not naturally generalize; ResNet50 visual TENs may generalize and amortize parameters, but add many parameters under sparse RL.Source: §7.2 · pp. 28–29

7.3 Experimental Details and Results

• Training used PPO with GAE and a KL reward term, in batches of 1,024 samples across 32 GPUs for 100 iterations.The value network was a separate LLM with its own TEN, initialized from Stage 2.Source: §7.3 · p. 29, Figure 17 · p. 30
• Stage 3 produced terrain-specific Sodaracers that performed well on their conditioned terrain but failed elsewhere, and usually matched or exceeded the best dataset- or pretrained-LLM samples.The models tended to produce a different Sodaracer for each terrain in the distribution.Source: §7.3 · p. 31, Figure 18 · p. 32, Figure 19 · p. 33
• Across tunnel and bumpy distributions, Stage 3 improved over Stage 2; it improved on dataset performance at 50% cutoff, whereas at 80% it was comparable to the dataset.The 80%-cutoff models tended to perform slightly better and train more stably, but differences were not significant.Source: §7.3 · p. 31, Figure 18 · p. 32, Figure 19 · p. 33

7.4 Qualitative Observations

• Qualitatively, Stage 3 produced short Sodaracers for tunnel and, in earlier ELM experiments, repeatedly found a wheel effective on bumpy terrain.The wheel did not re-emerge in the final experiments; the authors hypothesize conditional invention could support open-endedness by introducing objects that alter environments.Source: §7.4 · p. 34

8 Discussion and Conclusion

• The authors contrast nature’s functionally endowed first-cell seed with randomized EC starts, noting that arbitrary mutations to an already-formulated program are unlikely to help.They propose that human-readable Python programs could serve as more nature-like seeds.Source: §8 · p. 34
• ELM with MAP-Elites bootstraps hundreds of thousands of examples in a domain foreign to the initial LLM from human-written seeds, and that data is used to train LLMs.The paper says an invention pipeline confirms the generated data's validity.Source: §8 · pp. 34–35
• The paper argues that code-trained LLMs enable ELM-based GP to move beyond the raw code-search landscape and may reopen challenges in GP.The experiment points to implications for open-endedness, deep learning, and reinforcement learning; the authors also hope to motivate synergies between EC and deep learning.Source: §8 · p. 35
• LLM-based mutations can drive exploration with search algorithms such as MAP-Elites, and optimizing mutation operators on output quality during search appears to improve exploration.Search discoveries may be useful directly, such as wheels in Sodarace, or provide examples and optimize solutions in data-sparse domains; the authors also suggest searching over MAP-Elites or LLM architectures and hyperparameters.Source: §8 · p. 35
• Flatground-walker knowledge enabled a model to generate terrain-specialized walkers, illustrating informed leaps to new frontiers that could, if continued indefinitely, yield unbounded emergent complexity.The authors frame continually moving beyond familiar distributions as a challenge for open-endedness.Source: §8 · p. 35
• Whether the resulting model generalizes outside its training distribution remains open, since extrapolating learned principles may require further weight updates, potentially alongside continual evolution and RL.RL can use existing LLM knowledge to bootstrap new tasks, but extrapolation is harder.Source: §8 · p. 35

A Comparing Mutation Operators

• The mutation comparison uses Tiny GP node mutations, hand-tuned per problem, to fix bugs in one perturbation step and probe whether LLMs can make sophisticated movements through code.Trials perturb buggy parents and test child correctness; GP cannot use the plain-text docstring, unlike LLMs, which can use language-based comments to guide code evolution.Source: A Comparing Mutation Operators · p. 42
• The comparison tests 4-Parity for coordinated code changes and Quadratic for more ambiguous bugs, where the intended function does not imply a canonical form.4-Parity calculates the parity of a 4-bit sequence; Quadratic evaluates a quadratic function from a, b, c, and x.Source: A Comparing Mutation Operators · p. 43

A.1.1 4-Parity

• The 4-Parity benchmark returns 0 for even parity and 1 for odd parity by summing input bits modulo two.Its first four bugs rename b-prefixed variables to c-prefixed variables one at a time; the fifth changes modulus two to modulus three, and GP receives additional c-prefixed terminals.Source: §A.1.1 · p. 43

A.1.2 Quadratic

• The Quadratic benchmark computes a*pow(x,2)+b*x+c, with up to two bugs introduced by replacing plus operators with minus operators from left to right.Figure 20 compares diff and GP mutation as these bugs are added.Source: §A.1.2 · p. 44, Figure 20 · p. 44

A.2 Comparing GP and Diff Mutation

• Diff mutation outperformed GP mutation on the Quadratic task with both bug cases, and its performance was unaffected by the second bug.The appendix notes that 4-Parity comparisons appear in Figure 1 of the main text and that the Quadratic pattern differs in how performance degrades.Source: §A.2 · p. 44, Figure 20 · p. 44

A.3 Comparing API-based Mutations

• The API-based edit-mode and prompt-engineering mutations both outperformed the paper’s 300M diff model on the Quadratic task.The passage says this task provides additional evidence that multiple mutation options can build on the paper’s work.Source: §A.3 · p. 44, Figure 21 · p. 45

B Seed Source Code

• The appendix presents source code for the ELM seed programs and points to videos of the Sodaracers they represent.The video link is given in the appendix passage.Source: B Seed Source Code · p. 45

B.1 CPPN Seeds

• The two CPPN-like seeds differ in whether the CPPN query function can evolve: CPPN-Fixed excludes its source, while CPPN-Mutable includes it.The mutable seed’s query function creates a grid of points and functionally connects them.Source: §B.1 · p. 45, §B.1.2 · p. 46

B.1.1 CPPN-Fixed; B.1.2 CPPN-Mutable

• The CPPN-Fixed seed uses distance and phase functions and returns the constructed walker, while CPPN-Mutable includes the query_cppn function and connection logic.The fixed seed’s displayed code includes a distance-based connection condition; the mutable version iterates over grid points and adds muscles when its connection function permits.Source: §B.1.1 · p. 46, §B.1.2 · p. 46

B.2 Square Seed; B.3 Radial Seed

• The square seed builds a square body with distance muscles and active center prongs, while the radial seed approximates a circle and connects its points to the center with muscles.Both displayed seed implementations return a walker.Source: §B.2 · p. 47, §B.3 · p. 48

• The paper’s language models are based on the GPT-3 architecture.Source: C Model Architectures · p. 48
• Further descriptions of the architecture and hyperparameters can be found in Brown et al. [2].Source: C Model Architectures · p. 48
• Table 1 describes model architectures through parameter counts, layer counts, bottleneck widths, attention-head counts, and attention-head dimensions.Source: Table 1 · p. 49

D Seed Robustness

• High Stage 1 QD scores do not guarantee adaptation: Radial-seed products showed chaotic, initial-condition-sensitive dynamics and adapted poorly to novel terrains.Similar chaotic behavior appeared with the CPPN-Mutable seed trained with the pretrained diff model, suggesting QD score does not fully capture generalization or adaptation.Source: D Seed Robustness · p. 49
• Proposed seed-biasing strategies include restricting precise parameter settings, adding stochasticity, and incrementally adjusting seeds; preliminary parameter restrictions had mixed results.The authors also suggest expanding or changing QD score to better align with robust downstream performance.Source: D Seed Robustness · p. 49
• Fine-tuning on CPPN-Fixed products let later CPPN-Mutable runs retain its qualitative traits while outperforming it quantitatively.The fixed seed acted as “training wheels” for learning to modulate the CPPN encoding; incremental design, potentially with interactive evolution, may shape ELM outputs.Source: D Seed Robustness · p. 49

E Final Map Approach to Stage 2

• With only 13K examples, the final-map dataset yields lower QD scores than percentage-threshold models and unstable Stage 3 experiments, making it an unreliable RL starting point.It selects each run’s highest-quality solution for each discovered niche without normalizing performance across runs; differences across seeds can leave niches weak or undiscovered.Source: E Final Map Approach to Stage 2 · p. 50

F Source Code for Completion Targets

• ELM adds structure and reuses function calls in completion targets, while comments may be nonsensical, preserved from seeds, or stripped by MAP-Elites parsimony pressure.Examples include nested loops in Wheel, repeated loops in Galloper, and repeated calls to make sensor, a renamed and modified Square-seed function.Source: F Source Code for Completion Targets · p. 50

F.1 Wheel

• The Wheel completion code defines a circle, iterates to add joints and muscles, and returns the resulting walker.Source: §F.1 · p. 51

F.2 Galloper

• The Galloper code defines make_sensor, calls it repeatedly to assemble sides, adds connecting muscles, and returns the walker.Source: §F.2 · pp. 51–52

F.3 Runner

• The Runner source listing includes connect and phase functions and returns the walker.Source: §F.3 · p. 52

G.1 Blob (from CPPN Seed)

• The Blob listing defines connect and phase functions with coordinate-based expressions, then returns the walker.Source: §G.1 · p. 53

G.2 Hopper (from Square Seed)

• The Hopper listing constructs a square and adds muscles between sides and the center before returning the walker.Source: §G.2 · pp. 53–54

G.3 Centipede (from Radial Seed)

• The Centipede listing defines a circle with eccentricity 1.4 and adds muscles among its points before returning the walker.Source: §G.3 · pp. 54–55

H Probing Stage 2 Models

• Stage 2 probes added fitness and width, height, and mass comments, testing postfix fitness prediction and prefix-conditioned generation of walkers with requested properties.Source: H Probing Stage 2 Models · p. 55
• On held-out in-distribution examples, 300M prefix/postfix models consistently associated width, height, and mass, but fitness conditioning was less reliable because fitness depends on walker dynamics.Source: H Probing Stage 2 Models · p. 55
• A prefix model struggled to target held-out width and height bands, although its samples covered the holdout area, suggesting variation sufficient for interpolation or slight extrapolation.Source: H Probing Stage 2 Models · p. 56
• For Radial-seed Sodaracers unseen in training, postfix fitness prediction had Spearman correlation 0.08; larger models and more generated data were proposed, not tested, as possible remedies.Source: H Probing Stage 2 Models · p. 56
• Prefix-trained models did not outperform models without prefixes in the Appendix I interpolation setup, leaving open how Stage 2 could encode fitness or other information to benefit Stage 3 reinforcement learning.Source: H Probing Stage 2 Models · p. 56
• With 300M-parameter models and current training data, Stage 2 models learned Sodarace structure modestly but were not robust out of distribution, leaving open-endedness implications unclear.Source: H Probing Stage 2 Models · p. 56

I Interpolation Experiments

• The experiment varied tunnel heights to test whether a ResNet TEN could map taller tunnels to faster, taller Sodaracers and lower tunnels to shorter, slower ones.The design exploits the tradeoff that larger Sodaracers move faster on flat terrain but cannot navigate low tunnels.Source: I Interpolation Experiments · pp. 56–57
• The model repeatedly produced the same Sodaracer regardless of tunnel height, and that unconditional solution could not navigate every terrain in the distribution.This local optimum confounded the test of whether inventions can vary smoothly with the environment.Source: I Interpolation Experiments · p. 57
• The authors suggest RL favors simpler unconditional local optima, although the main experiments produced conditional solutions when no unconditional solution was sufficient.Because the tested solution was suboptimal, it remains unclear whether the pipeline can produce complex inventions that smoothly adapt to environmental changes.Source: I Interpolation Experiments · p. 57
• In more complex, open-ended processes where unconditional solutions are insufficient, conditional inventors might learn domain structure and achieve higher rewards.This is proposed as a future possibility, not a result of the interpolation experiment.Source: I Interpolation Experiments · p. 57
• Future work proposes supervised terrain–Sodaracer samples to fine-tune the LLM and TENs, unseen Stage 2 terrains in Stage 3, and replacing Stage 1 MAP-Elites with POET [40].The proposed supervised approach samples the unconditional LLM and uses the best-performing samples for each terrain; POET could generate such a dataset without hand-specified terrain distributions.Source: I Interpolation Experiments · pp. 57–58