ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution

Motivation and framework

• ShinkaEvolve addresses existing systems’ need for thousands of evaluations and closed-source implementations through three innovations that improve sample efficiency and enable community access.The innovations are adaptive parent and LLM sampling, code proposal novelty rejection-sampling, and bandit-based LLM ensemble selection; the complete implementation is released openly.Source: §1 · p. 2

Experimental scope

• ShinkaEvolve demonstrates innovation beyond human- and LLM-generated solutions across circle packing, AIME agentic design, ALE-Bench, and mixture-of-expert load-balancing loss design.The four domains cover mathematical optimization, agentic design, competitive programming, and LLM training design.Source: §1 · p. 2

Open-source release

• The framework releases implementation details under the Apache 2.0 license together with an interactive tool for monitoring the evolutionary search process.This open-source release is intended to support reproducibility and community-driven improvements.Source: §1 · p. 2

Evolutionary Code Optimization with LLMs

• LLM-guided evolutionary code optimization mutates, recombines, and archives generated programs across reward, preference, mathematical science, and other applications, including AlphaEvolve, OpenEvolve, and LLM4AD.These systems resemble population-based LLM-guided tree search, while ShinkaEvolve adds rejection sampling, LLM prioritization, and online meta-scratchpad drafting.Source: §2 · p. 2

Open-Ended Agentic Discovery

• Open-ended evolutionary principles combined with LLMs enable agentic systems capable of continuous innovation (Zhang et al., 2025).Source: §2 · p. 2
• Unlike explicit diversity metrics, LLM agents use learned representations to generate semantically coherent solutions and connect related programs across evolutionary generations.ShinkaEvolve combines stepping-stone solutions by having LLM agents generate mutations and evaluate program relationships, allowing successful patterns to propagate through recombination.Source: §2 · p. 3

3. Method

• ShinkaEvolve’s three-phase loop samples programs, proposes LLM-guided mutations, then executes them to guide ensemble selection and meta-scratchpad documentation.The phases combine parent and inspiration sampling, code novelty rejection, world feedback, and online knowledge diffusion.Source: §3 · p. 3

3.1. Parent and inspiration sampling

• Island populations preserve diversity through parallel evolution, elite archive maintenance, selective migration, and hierarchical sampling of parents, inspirations, and top-K programs.The island-specific best program cannot migrate, protecting subpopulation uniqueness while allowing knowledge diffusion across discovery substreams.Source: §3.1 · p. 3
• Parent selection spans uniform exploration, fitness-ranked exploitation, and performance–novelty weighting that favors high-performing programs with fewer offspring.Power-law sampling ranges from uniform selection to hill-climbing, while weighted sampling combines sigmoid-scaled performance with an inverse offspring-count novelty term; Figure 2 illustrates these strategies.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Program mutation and novelty assessment

• ShinkaEvolve samples models and parameters from a pool, then generates diversity through diff edits, full rewrites, and crossover mutations.Immutable code is protected with EVOLVE-BLOCK markers, while invalid patches trigger resampling with Reflexion parsing feedback (Shinn et al., 2024).Source: §3.2 · pp. 3–4
• Novelty rejection embeds mutable code, compares cosine similarity within islands, and invokes an LLM when similarity exceeds a threshold such as η=0.95.The framework combines a foundation-model ensemble with temperature sampling to encourage creative executable proposals; Figure 3 depicts the rejection process.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Execution and world feedback

• Execution archives scalar fitness, public metrics, and textual feedback, while UCB1-based adaptation rewards relative, bold improvements and periodic meta-scratchpads turn successful strategies into prompt guidance.LLM rewards are normalized for domain-scale invariance and evaluated against parent or initial-program baselines; recommendations are appended every T generations.Source: §3.3 · p. 5, Figure 4 · p. 5

4.1–4.4 Results

• ShinkaEvolve consistently improves solutions across four domains, from outperforming AlphaEvolve in circle packing within fewer than 150 evaluations to improving ALE-Agent by approximately 2.3% across 10 tasks.It also discovers generalizable AIME scaffolds and a new MoE load-balancing regularizer that improves results across training configurations.Source: Figure 5 · p. 6, §4.1 · p. 6, §4.3 · p. 8

4.1. Circle Packing: Reproducing & Improving AlphaEvolve Results

• ShinkaEvolve’s circle-packing solution combines goldenangle initialization, SLSQP refinement, simulated annealing, adaptive reheating, and constraint-aware perturbations to escape local optima.Its evolution proceeds through rapid improvement, sustained exploration, and final refinement through restarts, with successful innovations propagating through the population tree.Source: §4.1 · p. 6

4.2. AIME: Evolving Agent Scaffolds for Math Reasoning

• The evolved AIME scaffold uses three diverse expert solvers, low-temperature peer review, and zero-temperature synthesis, achieving its best performance with 7 LLM queries within the 10-query constraint.The scaffold generalizes across newer AIME problems and transfers successfully across gpt-4.1-mini, gpt-4.1, and o4-mini, while revealing a performance-efficiency Pareto frontier.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3. ALE-Bench: Evolving Programs for Combinatorial Optimization

• On ALE-Bench, ShinkaEvolve improves ALE-Agent solutions by approximately 2.3% on average across 10 tasks, including an ahc039 result that would rank second on AtCoder.For ahc039, caching and targeted edge moves improve performance from 2880 to 3140, while broader modifications often remain close to the initialization solution, suggesting potential overfitting.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4. LLM Training: Evolving Losses for Balanced and Effective Experts

• For MoE load balancing, ShinkaEvolve discovers a regularizer that supplements global-batch balancing with adaptive penalties for under-specialized experts below a minimum usage threshold.The penalty strengthens when routing entropy is low and vanishes after experts cross the threshold, addressing cases where aggregate balance masks barely used experts; evaluations span three λ values and seven benchmarks.Source: §4.4 · pp. 9–10

Impact of Parent Selection Strategies

• Weighted sampling consistently outperforms random search and hill climbing across all tasks, sustaining improvement while hill climbing plateaus and random search converges poorly.Weighted sampling balances exploration and exploitation by selecting parents according to fitness and offspring count.Source: §5 · p. 10

Impact of LLM Ensembling and Prioritization

• The bandit-based LLM ensemble significantly outperforms single-LLM and fixed-ensemble approaches by dynamically prioritizing models that improve fitness.The fixed ensemble samples uniformly, whereas the bandit strategy balances exploration of underused models with exploitation of effective ones.Source: §5 · p. 11

Impact of Code Embedding-Based Rejection Sampling

• Embedding-based rejection sampling substantially outperforms no rejection sampling by preventing redundant mutations, while an LLM novelty judge provides only marginal additional improvement.The embedding filter rejects proposals whose similarity scores exceed 0.95, indicating that embeddings already provide an effective novelty proxy without extra computational overhead.Source: §5 · p. 11

Summary

• ShinkaEvolve achieves state-of-the-art results across four domains, including circle packing with 150 evaluations, AIME reasoning scaffolds, ALE-Bench improvements, and novel mixture-of-expert load balancing.The circle-packing result represents an orders-of-magnitude improvement in evaluation efficiency.Source: §6 · p. 11

Limitations

• The implementation uses fixed configurations with limited automatic control over exploration-exploitation balance, which may vary across domains.Task specification also requires manual human expertise for objective functions and evaluation, and the framework is limited to problems with well-defined numerical objectives.Source: §6 · p. 11

Future Directions

• Automated LLM task generation, self-generated objectives, self-referential refinement, and online meta-learning are proposed to expand autonomy and enable continuous discovery improvement.These directions could support applications in unexplored domains and transition the framework toward true open-endedness.Source: §6 · p. 11

Broader Impact & Ethical Considerations

• The open-source release democratizes evolutionary optimization and exceptional sample efficiency reduces computational barriers, but large-scale LLM API costs may still constrain access.The impact is especially relevant for researchers and practitioners lacking proprietary systems and for resource-constrained environments.Source: §6 · p. 11

Author Contribution List

• Robert Tjarko Lange led the project, designed the core ShinkaEvolve codebase, implemented and collected results for Circle Packing, AIME, and ALE-Bench, and wrote the manuscript.Source: Author Contribution List · p. 12
• Yuki Imajuku helped establish the ALE-Bench infrastructure, advised on its results, and supported writing the ALE-Bench manuscript section.Source: Author Contribution List · p. 12
• Edoardo Cetin contributed to ShinkaEvolve design discussions, developed the adaptive LLM sampling method and Hydra configuration, ran LLM-training and Mixture-of-Experts evolution experiments, and co-wrote the manuscript.Source: Author Contribution List · p. 12

A. Shinka Implementation Details

• ShinkaEvolve sequentially generates program proposals and evaluates them through a job queue, with each proposal based on all completed jobs stored in the database.This queue-based design ensures proposals use the latest completed evaluation results.Source: A. Shinka Implementation Details · p. 19
• The fully asynchronous implementation improves throughput but can generate proposals before prior jobs finish, creating off-archiveness and execution-order effects from model query speed.The authors identify the trade-off among throughput, sample efficiency, and off-archiveness as an open research question.Source: A. Shinka Implementation Details · p. 19
• The Python API roughly adopts OpenEvolve’s high-level interface.The paper introduces the interface through a minimal ShinkaEvolve configuration and usage example.Source: A. Shinka Implementation Details · p. 19

• `run_shinka_eval` is the central evaluation call, receiving the program path and results directory while returning metrics, correctness, and errors.Source: evaluate.py - Evaluation Script · p. 19
• The script performs three evaluation runs and configures experiment arguments, metric aggregation, and optional validation through named callbacks.Source: evaluate.py - Evaluation Script · p. 19
• Aggregation converts results into a combined score, separates shinka-visible and shinka-invisible data, stores extra data, and returns text feedback.Source: evaluate.py - Evaluation Script · p. 19
• When executed directly, the script parses the program path and results directory before invoking `main`.Source: evaluate.py - Evaluation Script · p. 19

B.1. Circle Packing Problem

• The task places 26 circles inside a unit square, maximizing their summed radii while preventing overlap and boundary violations.All circles must remain fully contained within the square.Source: §B.1 · p. 20
• The main ShinkaEvolve run uses OpenEvolve’s verification script with 1 × 10−6 numerical slack.Solutions were additionally checked with AlphaEvolve’s exact verification code (Novikov et al., 2025).Source: §B.1 · p. 20
• Reducing every discovered circle radius by 1×10−8 makes the relaxed solution trivially exact while causing only a negligible objective decrease.The relaxed and exact sums are 2.635983099011548 and 2.6359828390115476, respectively.Source: §B.1 · p. 20
• The discovered packing is presented visually, while exact-verification evolution results show convergence behavior and solution quality over time.Figure 10 presents the discovered solution, and Figure 11 presents asynchronous evolution results for exact verification.Source: Figure 10 · p. 20, Figure 11 · p. 20
• 2.63597770931127 is the score of the replicated solution under AlphaEvolve’s exact verification code, supporting exact state-of-the-art discovery through post-processing.The relaxed-to-exact adjustment reduces the sum of radii from 2.635983099011548 to 2.6359828390115476, a relative change below 10^-6.Source: §B.1 · p. 20
• Performance benchmarks compare ShinkaEvolve with AlphaEvolve (Novikov et al., 2025), OpenEvolve, and LLM4AD, including Evolution of Heuristics configurations.OpenEvolve scores come from its official implementation and examples, while AlphaEvolve’s sum of radii comes from its paper.Source: §B.1 · pp. 20–21
• The implementation specifies a dedicated ShinkaEvolve hyperparameter configuration for the circle-packing task.The configuration is summarized in Table 1.Source: Table 1 · p. 21

B.2. AIME Math Reasoning Agentic Harness Detailed Task Description.

• The task evaluates scaffold designs on 30 challenging AIME 2024 problems, limiting each problem to 10 LLM queries for computational and cost efficiency.The base model is gpt-4.1-nano, and scaffold evolution runs for 75 generations.Source: §B.2 · p. 22
• Each candidate scaffold is evaluated across three independent runs on the complete question set to reduce the impact of stochastic LLM queries.After evolution, the discovered scaffolds are evaluated on AIME 2023 and 2025 problems to assess generalization and robustness across base agent language models.Source: §B.2 · p. 22

B.3. ALE-Bench Problems Detailed Task Description.

• ALE-Bench evaluation uses 10 LITE heuristic programming problems, with public-test scores as fitness and private-test scores reported for the best solution.Figure 7 also reports the maximum private-test score among the top five publicly scored solutions to assess generalization.Source: §B.2 · p. 23
• The average ALE-Bench solution score increases only from 1923.5 to 1927.0, providing no significant evidence of overfitting to public test cases.This comparison follows the ALE-Agent evaluation protocol and contrasts public-test fitness with private-test performance.Source: §B.2 · p. 23

B.4. Mixture-of-Experts Load Balancing Loss

• The MoE search pretrains a 556M-parameter model with 64 experts and 8 active experts per token on 2B FineWeb tokens, optimizing cross-entropy alongside load balance.The LBL term is weighted by λ = 0.01, and load imbalance is measured by L1 deviation from uniform token allocation.Source: §B.4 · p. 25
• ShinkaEvolve discovers a global-batch LBL augmentation with an additional regularization term targeting under-specialized experts.The formulation uses a normalized complement of routing entropy and a minimum usage threshold τ = 0.064/N_E.Source: §B.4 · p. 26
• Across training budgets and LBL coefficients, the discovered LBL appears to improve average task performance, final perplexity, and missrouted-token fraction over the original LBL.The comparison is reported in Figure 14, though the authors note that the evolution and test architectures were similar and the training budget limited.Source: Figure 14 · p. 26, §B.4 · p. 26

C.1. Circle Packing Problem

• The discovered circle-packing implementation constructs 26 circles using simulated annealing with local perturbations and ring rotations, followed by SLSQP refinement.The search uses 250 simulated-annealing iterations and a final polishing optimization.Source: §C.1 · pp. 28–30

C.2. AIME Math Reasoning Agentic Harness

• The AIME reasoning harness uses three experts within a seven-call budget, combining generation, review, and synthesis stages.It sets generation, review, and synthesis temperatures to 0.7, 0.1, and 0.0, respectively.Source: §C.2 · p. 31
• The harness extracts boxed numerical answers and falls back first to majority voting over reviewed answers, then over original solutions when synthesis fails.If no answer can be determined, the ultimate fallback returns boxed {000}.Source: §C.2 · pp. 31–32
• The AIME harness generates diverse step-by-step solutions with expert personas, independently critiques each solution, and synthesizes a corrected canonical answer.The synthesis explicitly compares proposed numerical answers, evaluates reliability using critiques, and constructs a clear final solution.Source: §C.2 · pp. 31–32
• Each forward pass is constrained to a maximum of 10 LLM calls, and experiments report accuracy, total cost, processed instances, call counts, and per-instance results.The evaluation wrapper returns these quantities from agent_evaluation.Source: §C.2 · p. 32

C.3.1. ALE-Bench LITE task: ahc039

• The solver models fish as typed integer points and implements orthogonal polygon perimeter, containment, and self-intersection checks.Polygon validity additionally enforces vertex, coordinate, perimeter, axis-parallel-edge, and nonzero-length constraints.Source: §C.3.1 · pp. 33, 34, 36
• A KD-tree stores subtree bounding boxes and mackerel/sardine counts, enabling pruned rectangular queries that aggregate fully contained subtrees.Partial overlaps are handled by checking the node point and traversing its children.Source: §C.3.1 · pp. 33–35
• The implementation includes both full and localized orthogonal-segment intersection routines to detect polygon self-intersections while skipping adjacent edges.Segment checks cover collinear overlap and horizontal–vertical crossings; localized checks restrict testing to critical edges.Source: §C.3.1 · pp. 35–36
• Initial polygon generation maps fish labels onto a 350×350 coarse grid and uses 2D Kadane’s algorithm to identify a best-scoring rectangular region.The implementation defines a minimal valid polygon default.Source: §C.3.1 · p. 37
• The implementation maintains geometric safety by clamping coordinates, preventing degenerate polygons, caching critical edges, and enforcing minimum polygon size during the search.Fallbacks restore a default polygon when initialization or subsequent states become invalid or degenerate.Source: §C.3.1 · pp. 38–39
• The solver initializes simulated annealing with a Kadane-derived polygon, cached perimeter and score state, structural-validity checks, and a tiny fallback polygon.Initialization rejects polygons with too few or duplicate vertices, self-intersections, or invalid structure before annealing begins.Source: §C.3.1 · pp. 38–39
• Coordinate proposals snap to guides derived from the current best polygon and fish locations, otherwise using time-shrinking random displacements with smaller late-stage steps.Dynamic guide snapping becomes more likely over time, while static guides and bounded random perturbations provide fallback exploration.Source: §C.3.1 · pp. 39–41
• Annealing uses a time-dependent temperature schedule and adaptive probabilities for targeted edge moves, edge moves, bulge additions, and simplification as polygon limits are approached.The method runs until the global time limit, lowers temperature aggressively near the end, and suppresses bulge moves near vertex or perimeter limits.Source: §C.3.1 · p. 39
• Targeted and ordinary edge moves update fish counts through rectangle-based score-delta queries, correcting signs for edge orientation and polygon winding.The implementation selects nearby or relevant edges, changes horizontal or vertical coordinates, and updates m_count and s_count incrementally.Source: §C.3.1 · p. 40

Discovered ahc039 solution

• The ahc039 solution uses simulated annealing to mutate polygons through bulge insertion and collinear-vertex simplification, then validates and accepts candidates by objective score.Bulge coordinates may snap to dynamic or static guides or use time-dependent random depths; candidates are checked for structural soundness, distinct vertices, and local self-intersection before simulated-annealing acceptance.Source: §C.3.1 · pp. 41–44

C.3.2. ALE-Bench LITE task: ahc025

• The QueryManager caches 1-versus-1 comparisons using normalized min/max keys and 1-versus-2 comparisons by keys, returning '=' when queries_made_ref >= Q.Source: §C.3.2 · p. 45
• Weight estimation samples and sorts pivots, infers pivot weights through comparisons and binary bracketing, then interpolates or extrapolates non-pivot weights.The estimator uses multiplicative comparison factors and maintains monotonicity in the resulting weights.Source: §C.3.2 · pp. 46–47
• Assignment optimization initializes greedily by placing heavier items into the lightest groups, then improves variance through targeted relocations and simulated annealing.Local search prioritizes moves from the heaviest group to the lightest, while annealing favors heavy-to-light relocations and occasional swaps.Source: §C.3.2 · pp. 48–49
• ShinkaEvolve discovered an ahc025 solution for the ALE-Bench LITE task.The solution is presented as Listing 5.Source: §C.3.2 · p. 51

C.4. Mixture-of-Experts Load Balancing Loss

• The discovered loss combines a routing mismatch term with an entropy-scaled penalty for experts whose normalized selection falls below a minimum-usage threshold.The threshold is min_threshold = 0.01 * (64.0 / num_experts), the penalty coefficient is 0.1, and the final loss adds this regularizer to the main loss.Source: §C.4 · p. 52