PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Baselines are Competitive with Code Evolution

Yonatan Gideoni, Sebastian Risi, Yarin Gal

arXiv:2602.16805v1cs.AIcs.LG

TL;DR

Simple baselines are rarely compared with more elaborate code evolution pipelines, leaving their relative performance insufficiently tested. The paper compares two baselines across mathematical bounds, agentic scaffolds, and machine learning competitions, finding that they match or exceed sophisticated methods. It also identifies search formulation, prompt knowledge, and scaffold evaluation as important factors in these settings.

  • Problem

    Many code evolution systems have design choices that are not ablated and are not compared with simple baselines, leaving their relative performance insufficiently tested.

    Source: §1 · p. 1

  • Method

    The paper introduces two simple program-search baselines and compares them with code evolution systems across three domains under matched constraints.

    Source: §1 · pp. 1–2

  • Results

    Across the tested settings, at least one simple baseline matches or exceeds a purpose-built code evolution pipeline under the same budget constraints.

    Source: §1 · p. 2, §7 · p. 7

  • Takeaways & Limitations

    Fair method comparisons should control for domain knowledge, verifiers, search spaces, and the constraint that limits each task.

    Source: §7 · p. 8

    The paper leaves open whether any code evolution agent can perform well universally beyond capabilities trained into its base model.

    Source: §7 · p. 8

Abstract

from arXiv · show

Code evolution is a family of techniques that rely on large language models to search through possible computer programs by evolving or mutating existing code. Many proposed code evolution pipelines show impressive performance but are often not compared to simpler baselines. We test how well two simple baselines do over three domains: finding better mathematical bounds, designing agentic scaffolds, and machine learning competitions. We find that simple baselines match or exceed much more sophisticated methods in all three. By analyzing these results we find various shortcomings in how code evolution is both developed and used. For the mathematical bounds, a problem's search space and domain knowledge in the prompt are chiefly what dictate a search's performance ceiling and efficiency, with the code evolution pipeline being secondary. Thus, the primary challenge in finding improved bounds is designing good search spaces, which is done by domain experts, and not the search itself. When designing agentic scaffolds we find that high variance in the scaffolds coupled with small datasets leads to suboptimal scaffolds being selected, resulting in hand-designed majority vote scaffolds performing best. We propose better evaluation methods that reduce evaluation stochasticity while keeping the code evolution economically feasible. We finish with a discussion of avenues and best practices to enable more rigorous code evolution in future work.

1. Introduction

The paper compares simple program-search baselines with more sophisticated code evolution methods across three domains, addressing a lack of ablations and simple-baseline comparisons. It finds that search-space formulation and prompt knowledge can matter more than the pipeline for mathematical bounds, while small validation sets can lead to suboptimal agentic scaffolds.

1. Introduction

  • The authors propose two simple baselines and compare them across mathematical bounds, agentic scaffolds, and machine learning competitions.The domains impose different constraints, including API budget, function evaluations, and wall-clock time.Source: §1 · p. 1
  • Under matched constraints, at least one simple baseline matches or exceeds a purpose-built pipeline; for bounds, formulation and prompt knowledge outweigh pipeline choice, while small scaffold datasets can select suboptimal designs.The paper also proposes economical practices for robust scaffold evaluation and discusses future directions.Source: §1 · pp. 1–2
  • For mathematical bounds, changing the problem formulation and using domain knowledge in prompts can matter more than the code evolution pipeline.Expert-designed search spaces influence the performance ceiling, while prompt knowledge can change search efficiency.Source: §1 · p. 2
  • For agentic scaffolds, small validation datasets contribute to high variance and selection of suboptimal scaffolds.The paper proposes evaluation practices intended to reduce stochasticity while keeping evaluation economical.Source: §1 · p. 2
  • The paper concludes with best practices and open problems, and releases code to support baseline comparisons and the recommended practices.Source: §1 · p. 2

2. Background

Code evolution uses language models to generate new program versions, often conditioning on previously selected programs. The paper places this approach in scientific discovery and other applications where solutions can be represented as programs.

2. Background

  • Code evolution typically starts from a program and uses an LLM to generate new versions, often selecting previous programs for the prompt based on diversity or fitness.The paper names AlphaEvolve, ShinkaEvolve, OpenEvolve, and ADAS as examples; some systems contain many components.Source: §2 · p. 2
  • In circle packing, a program can output circle centers and radii that are checked for validity, reducing the search to finding a suitable list of numbers.The objective is to maximize the radii sum while keeping circles non-overlapping and inside a unit box.Source: §2 · p. 2
  • Prior work notes that domain expertise or a changed search-space formulation can improve search, while ARC AGI methods often select the best program from many generated candidates.The authors do not compare their baselines with those ARC AGI pipelines.Source: §2 · pp. 2–3

3. Baselines for code evolution

The paper presents IID random sampling and sequential conditioned sampling as simple baselines, then sets comparison principles around task constraints and fairness. Sequential sampling is intended for iterative problems, whereas IID sampling selects the best program from an independently generated set.

3. Baselines for code evolution

  • IID random sampling prompts an LLM to generate programs, runs them, and selects the best by task outcome; sequential conditioned sampling modifies this approach for iterative improvement problems.The iterative approach is motivated by cases where initial LLM solutions fail to extend known constructions.Source: §3 · p. 3
  • Sequential conditioned sampling generates batches conditioned on a random subset of successfully run programs, without explicit fitness-based selection.Its longer prompts and reduced parallelism can be costly when wall-clock time is the main constraint.Source: §3 · p. 3

3.1. How should methods be fairly compared?

  • The paper treats solutions as equal when numpy.isclose finds them equal, accounting for numerical precision and small instabilities.Small numerical differences may still matter for tasks such as machine learning competitions or mathematical bounds.Source: §3.1 · p. 3
  • Fair comparisons use similar prompt knowledge across methods, while adjusting prompts when a method’s design requires it.The paper aims to keep prompts close to original-work prompts or minimally informative problem specifications.Source: §3.1 · p. 3

4. Finding mathematical bounds

Across nine mathematical-bound problems, simple baselines perform competitively with ShinkaEvolve, and changing a problem’s formulation can improve the bound more than changing the search pipeline. Results also show that prompt knowledge can help or hinder search, and that the baselines are faster in wall-clock time.

4. Finding mathematical bounds

  • IID RS matches or exceeds ShinkaEvolve on 4/9 problems and SCS on 6/9; SCS also matches or exceeds AlphaEvolve on 4/9 problems.These comparisons use a $20 budget; AlphaEvolve results are included for reference but excluded from comparisons because its budget is unknown and likely much higher.Source: §4 · p. 4, Table 1 · p. 4
  • At the $20 budget, both baselines compare favorably with ShinkaEvolve across budgets, and SCS generally outperforms it in average probability of matching or exceeding its result.The probability initially declines because ShinkaEvolve has a warmup period; the baselines take 1–3 hours per problem versus ShinkaEvolve’s approximately 10 hours.Source: Figure 2 · p. 4, §4 · pp. 4–5
  • For circle packing, IID RS found the 2.63598 score after sampling 1,000 programs when its prompt included a function for finding maximum radii given circle centers.This example shows that prompt domain knowledge can improve a method without changing its pipeline.Source: §4 · p. 4
  • For circle packing, ShinkaEvolve found 2.63598 in all three runs from a minimal initial program, while two of three runs with the original program found subpar packings.Combining the original program with a prompt describing typical good and bad solutions yielded the same bound in all three runs.Source: §4 · p. 4

4.1. A problem’s search space dictates its performance ceiling

  • When methods find similar bounds for a problem, they may share a performance ceiling, but it remains unclear whether those bounds are tight or similarly suboptimal.The authors argue that expert-designed formulations and search spaces matter more than the pipeline for improving bounds.Source: §4.1 · p. 5
  • Reformulating the uncertainty inequality changed its search space and led all methods to find 0.3482 at the same $20 budget, improving on AlphaEvolve’s 0.3521 bound.The authors state that this formulation change produced a larger improvement than optimizing a given setup, regardless of pipeline.Source: §4.1 · p. 5

5. Evolving agentic scaffolds is highly stochastic

Automated scaffold search is vulnerable to noisy evaluation on small validation sets, and its test performance does not surpass the manually designed majority-vote scaffold. Re-evaluation and an evaluation cascade offer more robust comparisons while controlling evaluation costs.

5. Evolving agentic scaffolds is highly stochastic

  • Automated search methods show >10% validation-to-test accuracy drops, while majority-vote baselines degrade by about 5%.AIME 2024 is the validation set and AIME 2025 the test set.Source: §5 · p. 5, Figure 3 · p. 6
  • Validation sets of about 100 questions can make scaffold selection noisy, so strong validation performance may be coincidental.Re-evaluating discovered scaffolds 10 times yields lower underlying validation accuracies than initially reported.Source: §5 · p. 5, Figure 3 · p. 6
  • With an evaluation cascade, IID RS selects a scaffold that generalizes better than those found without a cascade, but does not outperform majority vote@10.Figure 3 reports a 49.5% probability of improvement over majority vote@10 on the test set.Source: Figure 3 · p. 6, §5 · p. 6
  • Even with 10 evaluations per question, majority vote@5 accuracy on AIME 2025 has a standard deviation above 1%.The distributions are based on 100 sampled answers per question and bootstrapping.Source: Figure 4 · p. 6
  • For final reporting, use an effective test set of at least 300 questions and 95% confidence intervals; higher-variance datasets may need more samples.Here, 300 questions is equivalent to 10 repetitions.Source: §5 · p. 6
  • An evaluation cascade rechecks potentially competitive scaffolds on a larger validation set, while probability of dominance compares a method against a set of alternatives.The cascade reduces evaluation cost compared with simply increasing evaluations for every scaffold.Source: §5 · p. 6

6. Machine learning competitions

The paper evaluates simple baselines and AIDE on ten Kaggle competitions under a 24-hour limit. SCS performs similarly to AIDE, matching or exceeding it on six competitions.

6. Machine learning competitions

  • Each method had 24 hours per competition, with one RTX 8000 GPU and 12 CPU cores.The evaluation used the ten-competition subset of Schmidgall et al. (2025).Source: §6 · p. 7
  • AIDE is a Kaggle-focused code-evolution pipeline that prior work found outperformed other code-evolution systems on MLE bench.The cited comparison includes MLAB and OpenHands.Source: §6 · p. 7
  • SCS matches or exceeds AIDE on 6/10 Kaggle competitions, and the two methods perform similarly overall.Table 2 reports results across the ten-competition subset.Source: §6 · p. 7

7. Discussion

Across the three domains, simple baselines match or exceed sophisticated code evolution pipelines, while domain knowledge, problem formulation, and evaluation choices often matter more than pipeline complexity. The discussion recommends fair baseline comparisons and more reliable scaffold evaluation, while noting cost and task dependence as constraints.

7. Discussion

  • Across three domains, simple baselines match or exceed sophisticated pipelines; majority-vote scaffolds also outperform automated search systems.For mathematical bounds, domain knowledge and problem formulation affect search efficiency and its performance ceiling more than the pipeline.Source: §7 · p. 7
  • A scientific discovery does not establish that the method used to find it is performant; search methods need fair comparisons against simple baselines.The discussion distinguishes demonstrating a discovery from proposing a search method.Source: §7 · p. 7
  • Fair comparisons should align models, prompts, verifiers, search spaces, and budgets, including tuning costs in the search budget.Different prompts and domain knowledge can make apparent improvements stem from factors other than the pipeline.Source: §7 · p. 8
  • Code evolution is expensive and time-consuming, and pipeline improvements may help one setting while adding cost without substantial gains in another.The discussion also questions whether any current code evolution agent performs well universally beyond its base model’s capabilities.Source: §7 · p. 8
  • Changing a verifier changes the defined search space, and the discussion says such reformulations can improve bounds more than changing the pipeline.Although code evolution is theoretically Turing-complete, systems have not been observed to find new mathematical formulations in practice.Source: §7 · p. 8
  • For scaffold searches, evaluate enough samples during search and testing, use an evaluation cascade, and report probability-based comparisons with 95% confidence intervals.These practices aim to minimize stochasticity while keeping search economically feasible.Source: §7 · p. 8

Impact Statement

The impact statement warns that stronger code evolution may bring unknown societal effects and that rigorous evaluation can be costly. Its related-work overview places the paper among research on code evolution, evaluation robustness, and scaffold search.

Impact Statement

  • Automation may lead to job loss and other social problems that policymakers would need to address.The impact statement also notes that code evolution improvements may yield scientific discoveries alongside unknown, potentially negative societal effects.Source: Impact Statement · p. 8
  • More baselines and thorough evaluations can make code evolution research costly and less accessible to some communities.The paper points to Appendix N for ways to reduce costs while retaining scientific rigor.Source: Impact Statement · p. 8

A. Related work

B. AlphaEvolve problems

The appendix lists nine AlphaEvolve problems and their prior and AlphaEvolve bounds, then specifies the sampling and configuration choices used for mathematical-bound experiments.

B. AlphaEvolve problems

C. Finding bounds for math problems technical details

  • The baselines used Gemini 2.5 Pro with temperature 0.8, top-p 0.95, a 1024-token thinking budget, and a five-minute evaluation limit; these settings were not thoroughly tuned.Source: C. Finding bounds for math problems… · p. 11
  • IID RS sampled 2,000 programs, while SCS generated 1,200 across six trials and conditioned later generations on three successful prior programs.Each SCS trial used 10 generations of 20 programs.Source: C. Finding bounds for math problems… · p. 12
  • ShinkaEvolve was adjusted for a fairer comparison, including a 1,024-token thinking budget, five island subpopulations, and two archive programs for inspiration.The authors note that default settings were not competitive with the baselines and would otherwise use a larger thinking budget.Source: C. Finding bounds for math problems… · p. 12

D. OpenEvolve results on math problems

OpenEvolve often crashed with Gemini and lacked budget controls; its successful runs ranked behind the baselines and ShinkaEvolve on average. The appendices specify the circle-packing prompts and bootstrap procedures used to assess comparisons.

D. OpenEvolve results on math problems

E. Different circle packing prompts

  • The circle-packing prompt asks for 26 non-overlapping circles inside the unit square, maximizing the sum of their radii.The supplied verifier checks that circles are disjoint and fully inside the square.Source: E. Different circle packing prompts · pp. 12–13

F. Bootstrap estimate

  • The IID bootstrap estimates matching or exceeding ShinkaEvolve by repeatedly sampling k programs from 2,000 generated programs and checking whether any meet its score.The SCS bootstrap exhausts generations within a trial under budget, then continues to another trial; Table 1 used chronological trial order after exhaustion.Source: F. Bootstrap estimate · p. 14
  • Confidence intervals come from bootstrapping over sampled programs or trials; the authors say estimates may be overly negative and have asymmetric, low lower bounds.They attribute this pattern to counterfactual information being available for worse outcomes but not better ones, alongside the nonlinear probability calculation.Source: F. Bootstrap estimate · p. 14

G. Figure 2 per-problem Breakdown

  • Figure 5 plots each problem’s probability that a baseline matches or exceeds ShinkaEvolve, with shaded 95% confidence intervals.Source: Figure 5 · p. 15

H. Math bounds per-program evaluation sample efficiency

The appendices compare mathematical-bound methods by evaluated programs, describe a broader uncertainty-inequality formulation, and document evaluation setups and comparison metrics. They also give implementation details and cost-conscious development options.

H. Math bounds per-program evaluation sample efficiency

  • ShinkaEvolve’s slightly lower cost per program makes the baselines’ matching-or-exceeding probabilities in Figure 6 slightly lower than under API-budget comparisons.Figure 6 aggregates across nine problems as a function of the fraction of programs evaluated; Figure 7 gives per-problem probabilities.Source: H. Math bounds per-program evaluation… · p. 15, Figure 6 · p. 16

I. Improved uncertainty inequality formulation

  • The uncertainty-inequality formulation uses probabilist’s Hermite polynomials to avoid numerical instability and retains coefficients through α7 to cover a larger function class.Going beyond α7 might further reduce the bound, but the authors encountered numerical instabilities, likely from very high-order polynomials.Source: §I · p. 16

J. Agent evolution technical details

  • For scaffold experiments, all methods used Gemini-2.5 Pro, scaffolds were limited to 10 LLM calls per question, and each was evaluated three times on AIME 2024’s 30 questions.The authors report that ADAS with Gemini-2.5 Pro often produced pipelines that failed, including by attempting plots or evaluating code its evaluation agent would produce.Source: J. Agent evolution technical details · pp. 16–17

K. Probabilities of improvement and dominance

  • Probability of dominance can be estimated efficiently by Monte Carlo, but an individual method’s probability generally falls as more methods are compared; 300 questions is suggested as a sensible minimum for expensive evaluations.The authors recommend stricter evaluation cascades and independent reruns as scaffold stochasticity increases.Source: K. Probabilities of improvement and… · p. 18

L. MLE bench technical details

N. Tips for economically feasible code evolution development

  • Development costs can be reduced by using cheaper or open-source LLMs and by using evaluation cascades; open-source models may be less capable and require rented GPUs.The authors note that laptop-CPU MLE Bench experiments can avoid GPUs while limiting the available toolset.Source: N. Tips for economically feasible code… · pp. 18–19

M. Full MLE bench competition names

  • Table 5 lists the full competition names corresponding to abbreviations in Table 2.Source: Table 5 · p. 19

O. Limitations

The authors identify limited comparability as the paper’s main limitation, alongside run costs and constraints in comparison scope. They also qualify the MLE Bench comparison against AIDE.

O. Limitations

  • Comparisons are limited because many pipelines target specific tasks, some are not open-source, and inconsistent setups make new comparisons effortful and performance drops hard to attribute.The authors argue that greater standardization across benchmarks and tasks would enable better comparisons.Source: O. Limitations · p. 19
  • Code-evolution run costs constrain the number of economically feasible experiments, and evaluation costs lead most experiments to use Gemini models.The cost-reduction approaches discussed by the authors are not yet standard in the community.Source: O. Limitations · p. 19
  • The authors do not compare improvement magnitudes for mathematical bounds or MLE competitions, instead using pairwise comparisons whose individual-task results are less meaningful.For circle packing, an SCS–ShinkaEvolve difference of about 10^-4 can be meaningful near the best bound but otherwise meaningless.Source: O. Limitations · p. 19
  • AIDE is more complicated than the baselines but simpler than some other code-evolution systems, and the authors know of no more complicated, more performant MLE Bench method.They compare against AIDE because it is well known.Source: O. Limitations · p. 19
Loading 2602.16805v1…