Source-linked AI summary
ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
Robert Tjarko Lange, Yuki Imajuku, Edoardo Cetin
TL;DR
Existing LLM-based code-evolution methods are sample-inefficient and often closed-source, limiting efficient and broad scientific discovery. ShinkaEvolve addresses this with exploration–exploitation-aware parent sampling, novelty rejection, and adaptive LLM ensembles. Across four domains, it reports state-of-the-art results, including a circle-packing solution using 150 evaluations, while remaining constrained by manual task specification and numerical objectives.
Problem
Existing code-evolution approaches require thousands of samples to find effective solutions and remain closed-source, restricting broad adoption and extension.
Method
ShinkaEvolve combines parent sampling, code novelty rejection-sampling, and adaptive performance-based LLM ensemble selection in an open-source evolutionary framework.
Results
ShinkaEvolve achieves state-of-the-art results across four domains, including circle packing with 150 evaluations, AIME reasoning scaffolds, ALE-Bench improvements, and novel mixture-of-expert load-balancing losses.
Takeaways & Limitations
The framework demonstrates broad applicability and sample-efficient open-ended discovery across mathematical optimization, agentic design, competitive programming, and LLM training design.
Takeaways & Limitations
The framework requires manual human expertise for objective functions and evaluation and is limited to problems with well-defined numerical objectives.
Abstract
from arXiv · showhide
We introduce ShinkaEvolve: a new open-source framework leveraging large language models (LLMs) to advance scientific discovery with state-of-the-art performance and unprecedented efficiency. Recent advances in scaling inference time compute of LLMs have enabled significant progress in generalized scientific discovery. These approaches rely on evolutionary agentic harnesses that leverage LLMs as mutation operators to generate candidate solutions. However, current code evolution methods suffer from critical limitations: they are sample inefficient, requiring thousands of samples to identify effective solutions, and remain closed-source, hindering broad adoption and extension. ShinkaEvolve addresses these limitations, introducing three key innovations: a parent sampling technique balancing exploration and exploitation, code novelty rejection-sampling for efficient search space exploration, and a bandit-based LLM ensemble selection strategy. We evaluate ShinkaEvolve across diverse tasks, demonstrating consistent improvements in sample efficiency and solution quality. ShinkaEvolve discovers a new state-of-the-art circle packing solution using only 150 samples, designs high-performing agentic harnesses for AIME mathematical reasoning tasks, identifies improvements to ALE-Bench competitive programming solutions, and discovers novel mixture-of-expert load balancing loss functions that illuminate the space of optimization strategies. Our results demonstrate that ShinkaEvolve achieves broad applicability with exceptional sample efficiency. By providing open-source accessibility and cost-efficiency, this work democratizes open-ended discovery across diverse computational problems.
1. Introduction
ShinkaEvolve targets sample inefficiency and limited accessibility in LLM-driven code evolution. It introduces an open-source evolutionary framework with three algorithmic innovations and validates it across four problem domains.
- Motivation: Existing LLM-driven scientific-discovery systems use language models as mutation operators in iterative candidate refinement.These systems have been applied to competitive programming, mathematical optimization, and automated agentic design.
- Contributions: ShinkaEvolve improves sample efficiency through parent sampling, code novelty rejection-sampling, and adaptive performance-based LLM ensemble selection.The framework combines these three mechanisms as its core algorithmic contribution.
- Contributions: The framework is evaluated on circle packing, AIME agentic design, ALE-Bench competitive programming, and mixture-of-expert load-balancing loss design.The authors also release the implementation under the Apache 2.0 license with an interactive search-process visualization tool.
2. Related Work
Prior work combines LLMs with evolutionary code optimization and open-ended discovery to generate and refine computational solutions. ShinkaEvolve builds on this lineage while using learned representations and evolutionary structures to support semantic novelty and innovation.
- Evolutionary Code Optimization with LLMs: Evolutionary code optimization mutates and recombines previously generated code to optimize reward, preference, mathematical-science, and other objectives.LLMs can serve as recombination engines within this process.
- Open-Ended Agentic Discovery: Integrating LLMs with open-ended evolutionary principles produces agentic systems capable of continuous innovation.This work frames open-ended discovery as a related direction for LLM-based evolutionary systems.
- Open-Ended Agentic Discovery: LLM agents use learned representations to generate creative, semantically coherent solutions and organize mutations into evolutionary trees across generations.ShinkaEvolve also combines stepping stones, or suboptimal intermediate solutions, as building blocks for breakthrough innovations.
3. Method
ShinkaEvolve evolves programs through archive-based sampling, LLM-guided mutation, execution feedback, and adaptive search control. Its method combines exploration–exploitation-aware parent selection, novelty rejection, ensemble adaptation, and online textual refinement.
- Algorithm Overview: The control flow samples parents and inspirations, proposes LLM-guided mutations, then executes programs to obtain feedback for subsequent evolution.The archive uses island subpopulations, while execution feedback updates ensemble probabilities and supports meta-scratchpad drafting.
- Parent and inspiration sampling: The archive stores evaluated programs, fitness scores, metadata, and diverse parent or inspiration exemplars for creative recombination.An elite-size constraint and island model structure organize the retained search population.
- Parent and inspiration sampling: Parent selection balances performance and novelty by weighting programs according to fitness-derived scores and offspring counts.Weighted sampling favors strong programs while giving preference to programs with fewer offspring; power-law sampling spans uniform exploration to hill-climbing exploitation.
- Program mutation: Mutations use sampled LLMs and parameters through diff-based edits, full rewrites, or crossover mutations.Immutable code blocks are protected during rewrites, and invalid proposals are resampled with parsing feedback.
- Novelty assessment: Novelty rejection samples code embeddings, compares mutable snippets across island programs, and invokes an LLM novelty judge when similarity exceeds a threshold such as η=0.95.The procedure combines embedding-based screening with semantic assessment before execution.
- Execution and feedback: Execution produces scalar fitness, public metrics, and textual feedback that are stored in the archive for future mutation prompts.The prompts expose current code, performance metrics, and feedback to guide targeted improvements.
- Adaptive LLM sampling: A UCB1-based ensemble adapts LLM sampling probabilities using visitation counts and mutation rewards relative to parent and initial-program baselines.The reward transformation emphasizes relative improvement across a non-stationary archive and promotes bold mutations.
- Online refinement: Every T generations, the meta-scratchpad summarizes evaluations and turns recurring strategies into recommendations appended to mutation prompts.It maintains individual program summaries, global insights, and implementation recommendations.
4. Results
Across circle packing, AIME agent scaffolds, ALE-Bench, and MoE loss design, ShinkaEvolve finds strong solutions and improvements with compact evolutionary searches.
- 4.1. Circle Packing: 150 generations demonstrate sample-efficient circle-packing search versus approaches typically requiring thousands of evaluations.The search progresses through rapid improvement, sustained exploration, and final convergence, with successful innovations propagating through evolutionary branches.
- 4.1. Circle Packing: The evolved circle-packing algorithm combines golden-angle initialization, SLSQP refinement, simulated annealing, and adaptive perturbations to escape local optima.Constraint-aware radius computation maintains feasibility while reheating prevents premature convergence.
- 4.2. AIME: AIME scaffold evolution discovers a performance–query-budget Pareto frontier, with maximum performance at 7 LLM queries and transfer across models.The discovered scaffold generalizes to 2023 and 2025 AIME problems and adapts to gpt-4.1-mini, gpt-4.1, and o4-mini.
- 4.3. ALE-Bench: Across 10 ALE-Bench tasks, ShinkaEvolve improves ALE-Agent solutions by approximately 2.3% on average.On ahc039, the solution moves from 5th to 2nd place on the AtCoder leaderboard if it had participated.
- 4.4. LLM Training: ShinkaEvolve discovers an MoE load-balancing loss that adds adaptive regularization for under-specialized experts and avoids over-regularizing balanced layers.The term activates below a minimum usage threshold and strengthens when routing entropy is low, addressing a blind spot of global-batch balancing.
5. Ablations & Analysis
Ablations show that each major ShinkaEvolve component improves search: weighted parent sampling sustains progress, adaptive LLM selection raises performance, and novelty rejection avoids redundant mutations.
- Parent Selection: Weighted parent sampling consistently outperforms random search and hill climbing across all tasks.Hill climbing performs strongly initially but plateaus, whereas weighted sampling maintains steady improvement.
- LLM Ensembling: Bandit-based LLM ensembling significantly outperforms single-LLM and fixed-ensemble approaches.The adaptive strategy prioritizes models according to their contributions to fitness improvements.
- Novelty Filtering: Embedding-based rejection sampling substantially improves performance over accepting every proposal by preventing redundant mutations.The approach rejects proposals whose embedding similarity exceeds 0.95.
- Novelty Filtering: An additional LLM novelty judge provides only marginal improvement beyond embedding-based filtering.This suggests embedding similarity is already an effective novelty proxy without extra computational overhead.
6. Discussion
ShinkaEvolve reports state-of-the-art results across four domains, while its current use requires manual task specification and remains limited to well-defined numerical objectives.
- Discussion: ShinkaEvolve achieves state-of-the-art results across circle packing, AIME reasoning, ALE-Bench, and mixture-of-expert load balancing.The circle-packing result uses 150 evaluations and is described as an orders-of-magnitude improvement.
- Limitations: The framework uses fixed configurations with limited automatic control over the exploration-exploitation balance, which may vary across domains.The limitation concerns adapting search control to differing task requirements.
- Limitations: Task specification requires manual human expertise for objective functions and evaluation.This constrains autonomy in setting up new discovery problems.
- Limitations: The framework is constrained to problems with well-defined numerical objectives.This limits applicability to more diverse evaluation domains.
- Future Directions: Future directions include automated task generation, self-generated objectives, self-referential refinement, and online meta-learning.These directions target greater autonomy and continuous improvement in discovery.
- Broader Impact: Open-source release and sample efficiency may reduce access barriers, although large-scale LLM API costs can still create economic barriers.The discussion identifies cost as a potential constraint on democratization.
Author Contribution List
The listed contributors divide responsibility across core framework leadership, ALE-Bench infrastructure and analysis, and adaptive LLM sampling plus mixture-of-experts experiments.
- Contributions: Robert Tjarko Lange led the project, designed the core codebase, conducted three domain evaluations, and wrote the manuscript.His listed evaluations covered Circle Packing, AIME, and ALE-Bench.
- Contributions: Yuki Imajuku helped establish ALE-Bench infrastructure, advised on results, and supported the ALE-Bench manuscript section.
- Contributions: Edoardo Cetin implemented adaptive LLM sampling and Hydra configuration, led LLM-training and MoE experiments, and co-wrote the manuscript.
A. Shinka Implementation Details
ShinkaEvolve uses a sequential proposal-and-evaluation queue, while an explored asynchronous design could increase throughput at the cost of off-archiveness.
- Implementation: LLMs generate program proposals sequentially, which are then added to a job evaluation queue.Each proposal uses all completed jobs stored in the database as context.
- Implementation: A fully asynchronous implementation was explored to increase throughput.Using separate job and proposal queues introduces a degree of off-archiveness.
- Implementation: Listing 1 provides a minimal ShinkaEvolve configuration and usage example.
evaluate.py - Evaluation Script
The evaluation script runs candidate programs repeatedly, aggregates their results, and exposes combined scores and feedback for ShinkaEvolve.
- The script imports run_shinka_eval to evaluate a program at a specified path and results directory.
- Three evaluation runs are configured to aggregate performance across repeated executions.The evaluator uses experiment and aggregation callbacks, with optional validation.
- Experiment parameters are supplied through a get_kwargs function for each run.
- The aggregation function returns a combined score, public and private metadata, extra stored data, and text feedback.
- A main entry point connects command-line program and results-directory arguments to the evaluation routine.
B.1. Circle Packing Problem
The circle-packing benchmark places 26 non-overlapping circles inside a unit square while maximizing their total radius. ShinkaEvolve obtains a high-quality solution through relaxed verification followed by exact validation.
- Circle Packing Problem: 26 circles must fit fully inside a unit square without overlap while maximizing the sum of their radii.The task combines discrete placement with continuous radius optimization and contains multiple local optima.
- Verification: 2.635983099011548 is the relaxed-verification sum of radii, reduced to 2.6359828390115476 after shrinking each radius by 1×10^-8 for exact feasibility.The relative change is less than 10^-6.
- Results and Configuration: The section presents the discovered solution, its asynchronous evolution behavior, performance-score context, baseline comparisons, and task configuration.
- Verification: 2.63597770931127 is the score obtained when the discovered solution is replicated with AlphaEvolve’s exact verification code.The exact formulation requires more evaluated samples than the surrogate relaxed task.
- Verification: Surrogate relaxed evolution can be followed by post-processing to obtain an exact state-of-the-art circle-packing solution.
B.2. AIME Math Reasoning Agentic Harness Detailed Task Description.
ShinkaEvolve evolves agent scaffolds for AIME mathematical reasoning under a query and generation budget, then tests them on additional years and base models.
- Task Description: The AIME 2024 evaluation contains 30 competition-level mathematical reasoning problems.
- Evaluation Setup: Each problem permits at most 10 LLM queries, and scaffold designs evolve for 75 generations using gpt-4.1-nano.
- Evaluation Setup: Each candidate is evaluated across three independent runs on the complete question set to address stochasticity in LLM queries.
- Evaluation Setup: Discovered scaffold designs are evaluated on AIME 2023 and 2025 problems to assess generalization and robustness across base agent language models.
- Configuration: The supplied materials identify hyperparameter configurations for the math-reasoning harness and ALE-Bench problems.
B.4. Mixture-of-Experts Load Balancing Loss
This section introduces the MoE load-balancing problem, the experimental setup for evolving LBLs, and ShinkaEvolve’s discovered regularized loss. Additional results indicate improvements over global-batch LBL across performance and efficiency measures, while noting limited architectural diversity and training budget.
- Task and motivation: MoE layers activate only K=8 of 64 experts per token, allowing experts to specialize while requiring load balancing to prevent uneven routing.The evaluated model has 556M parameters, with 82M sparsely activated parameters per forward pass excluding embeddings.
- Existing load-balancing loss: The global-batch LBL aligns expert token usage with the router’s average soft assignment probabilities.It is described as one of the most widely adopted LBL designs.
- Evolution setup: ShinkaEvolve evolves LBL programs by minimizing cross-entropy together with an LBL-weighted load-imbalance objective.Cross-entropy is averaged over the last 10M tokens before forming the negated fitness score.
- Discovered loss: ShinkaEvolve discovers an augmentation of global-batch LBL with an additional regularization term targeting under-specialized experts.The formulation uses selection frequency, average router probabilities, normalized routing-entropy complement, and a minimum usage threshold.
- Additional results: Across LBL coefficients, the discovered loss appears to improve on the original across average task performance, final perplexity, and missrouted-token fraction.The authors caution that the evolved and tested architectures were similar and the training budget remained limited.
C.3.1. ALE-Bench LITE task: ahc039
The ahc039 solution combines geometric utilities with a KD-tree to represent points, test polygon containment, and accelerate rectangular fish-count queries.
- The solution represents geometric points with integer x and y coordinates and supports ordering, equality, and subtraction.
- Fish are modeled by coordinates and type, distinguishing mackerel from sardines for score computation.
- A KD-tree stores subtree bounding boxes and fish-type counts, enabling rectangle queries to prune disjoint regions and aggregate whole subtrees.
- The solution computes polygon perimeter with Manhattan edge lengths and counts fish contained within valid polygons.
- Polygon containment uses boundary checks followed by a winding-number test to classify points as inside or outside.
C.3.2. ALE-Bench LITE task: ahc025
The ahc025 solution estimates item weights under query limits using cached comparisons, pivot-based sorting, and binary-search bracketing.
- The QueryManager caches pairwise and three-item comparison results while tracking the number of queries made.
- Single-pivot estimation assigns weights relative to a base weight using comparison outcomes and separate greater-than and less-than factors.
- The method selects a bounded number of shuffled pivots and sorts them through comparison queries.
- Multi-pivot estimation brackets each new pivot with binary-search comparisons and derives an estimate from lower and upper bounds or a fallback factor.
- Non-pivot items are located relative to sorted pivots and receive exact or interpolated weights, with monotonicity enforced afterward.
- The discovered implementation is identified as a ShinkaEvolve ahc025 solution.