BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

Yangzhen Wu; Aaron J. Li; Wenjie Ma; Li Cao; Ziheng Zhou; Mert Cemri; Shu Liu; Yuran Xiu; Chenxiao Yan; Haikun Zhao; Bin Yu; Ion Stoica; Dawn Song

Source-linked AI summary

BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution

Yangzhen Wu, Aaron J. Li, Wenjie Ma, Li Cao, Ziheng Zhou, Mert Cemri, Shu Liu, Yuran Xiu, Chenxiao Yan, Haikun Zhao, Bin Yu, Ion Stoica, Dawn Song

arXiv:2606.01286v1May 31, 2026cs.SEcs.AIcs.CLcs.LG

TL;DR

Benchmark saturation limits the ability of coding datasets to distinguish frontier models and provide training signal. BenchEvolver evolves reference solutions to create harder, verifiable task variants, producing substantially more challenging tasks across LiveCodeBench and SciCode and restoring benchmark discrimination.

Problem
Coding benchmarks are increasingly saturated, limiting their ability to measure frontier-model progress, distinguish capabilities, and guide training.
Method
BenchEvolver mutates reference solutions first, then derives task statements, examples, and tests from executable oracles with consistency and difficulty checks.
Results
BenchEvolver produces valid, diverse, substantially harder tasks across LiveCodeBench and SciCode, and LiveCodeBench-Plus better discriminates among frontier models.
Takeaways & Limitations
Evolved tasks can serve as harder evaluation challenges and support closed-loop self-improvement through reinforcement learning, including for the models that generated them.
Takeaways & Limitations
Scaling self-improvement requires controlling task difficulty, preserving algorithmic diversity, and balancing original with evolved tasks to avoid uninformative or overfit curricula.

Abstract

from arXiv · show

The rapid progress of frontier large language models has led to widespread benchmark saturation, limiting the ability of existing datasets to differentiate model capabilities or provide useful training signal. For instance, on LiveCodeBench, frontier models achieve over 99% Pass@1 on easy splits and exceed 90% Pass@1 on average across difficulty levels. Constructing new, challenging datasets typically requires substantial human effort, creating a bottleneck for progress. We introduce BenchEvolver, a solution-centric evolutionary framework that automatically transforms existing coding problems into harder variants. Rather than generating problems from scratch, BenchEvolver evolves reference solutions through structured transformations and derives corresponding statements and tests from the evolved solutions. This design grounds generation in executable semantics, enabling scalable construction of high-quality, diverse, and difficult tasks with verifiable correctness. Applying BenchEvolver to LiveCodeBench and SciCode, we obtain evolved tasks that are substantially harder while maintaining validity, reference correctness, and diversity. We further curate LiveCodeBench-Plus, a 91-problem benchmark combining evolved and difficult original LCB-v6 tasks, where frontier-model Pass@1 ranges from 27.5% to 62.6%, restoring clear discrimination among strong coding models. Importantly, evolved tasks remain challenging even for the model that generates them, enabling self-improvement. We further show that RL on evolved LCB tasks improves held-out coding performance: for gpt-oss-20b, seed+evolved training achieves +8.7 and +8.3 Pass@1 gains on LCB v6 Hard and LCB-Pro Easy, exceeding seed-only gains by 70.7% and 34.8%, respectively. Our results show that BenchEvolver can convert saturated benchmarks into frontier-level evaluation suites and reusable training signal.

1 Introduction

BenchEvolver addresses benchmark saturation by evolving executable reference solutions into harder, verifiable coding tasks rather than generating problems from scratch. Applied to LiveCodeBench and SciCode, it produces valid, diverse tasks and supports both more discriminative evaluation and model improvement.

Motivation: Over 99% pass rate on LiveCodeBench’s newest easy split and over 90% on average across difficulty levels leave benchmarks with diminishing discriminative power.Saturation also reduces useful training and analysis signal.
Motivation: Human construction of new benchmarks is expensive and difficult to scale, creating a bottleneck for continuous model improvement.Synthetic generation is a natural alternative, but existing pipelines often follow an asymmetric teacher–student paradigm.
Method: BenchEvolver mutates reference solutions first, then derives statements, examples, and tests from the evolved solution as an executable oracle.Accepted mutations must alter solution structure enough to make the parent algorithm insufficient.
Results: On LiveCodeBench and SciCode, BenchEvolver generates valid, diverse, substantially harder tasks and supports the difficulty-upgraded LiveCodeBench-Plus benchmark.LiveCodeBench-Plus combines validated evolved tasks with challenging original LiveCodeBench problems to improve discrimination among frontier models.
Results: For gpt-oss-20b, training on seed+evolved or evolved-only tasks improves held-out coding performance more than training on the original seed set.These results indicate evolved tasks can provide reusable RL signal in addition to serving as harder benchmark items.

2 Related Work

BenchEvolver relates to research on synthetic coding data, self-play, and evolutionary search. Prior work has synthesized diverse coding resources and used model-in-the-loop generation for self-improvement.

Prior work synthesizes code instructions, reasoning traces, repository-level bug-fix tasks, competitive-programming problems, and reinforcement-learning curricula to improve code models.
A complementary research line uses model-in-the-loop data generation to support self-improvement.
Examples include self-challenging tool-use agents and solver–conjecturer self-play.

3 Method

BenchEvolver is a closed-loop evolutionary framework that transforms executable seed programming tasks into harder, verifiable, and diverse variants. It evolves reference solutions first, derives statements and tests from them, validates consistency, selects by target-model failure, and uses memory to guide subsequent search.

Task representation: BenchEvolver represents each executable task through its statement, reference implementation, hidden tests, and execution harness, with the harness as the only domain-specific component.The framework supports stdin–stdout competitive-programming tasks and function-level scientific coding tasks with assertion tests and oracle artifacts when available.
Evaluation and selection: Difficulty is measured by executable target-model failure: an attempt succeeds only when it passes all hidden tests, and candidates are accepted only when their pass rate improves over the seed difficulty.The evaluator also filters ambiguity, misleading I/O, underspecified constraints, unnatural edge cases, and near-duplicate reskins, using bounded repair before rejection.
Solution-centric proposal: The Proposer mutates a parent reference solution into C′, then derives the statement, public examples, and hidden tests around the evolved executable behavior.This solution-centric direction is intended to introduce a dominant algorithmic lift that makes the parent approach insufficient.
Evaluation and selection: Candidate validity is established before difficulty measurement through benchmark-specific consistency checks among the statement, reference solution, tests, and harness.Competitive-programming validation triangulates the evolved reference solution, a statement-only brute-force solver, and a statement-only public-output oracle.
Memory and evolutionary loop: Local and global memories make the process adaptive by recording lineage outcomes, model error patterns, and mutation families, thereby avoiding repeated failures and enforcing diversity across lineages.When a mutation family has already succeeded elsewhere, a new candidate from that family must provide a larger difficulty gain to be accepted.

4 Experiments

Experiments show that BenchEvolver produces valid, harder, and more diverse coding tasks across competitive and scientific coding, constructs the 91-problem LIVECODEBENCH-PLUS benchmark, and supplies useful reinforcement-learning signal. Evolved tasks challenge both target models and their generating models, while seed+evolved training performs best on public held-out settings.

Task evolution: BenchEvolver reliably transforms saturated LiveCodeBench seeds into valid evolved tasks, with the full solution-centric method outperforming the problem-centric baseline in coverage and post-hoc validity.The memory-free ablation also underperforms, indicating that accepted lineages and prior failures guide search beyond independent one-step mutations.
Task evolution: Evolved tasks substantially reduce target-model pass rates across lightweight and frontier models, difficulty levels, and evolver models, including clear accuracy drops for each evolver on its own tasks.This consistency indicates the difficulty is not an artifact of one generator or prompt configuration.
Task evolution: 19 to 30: evolved problems expand the number of distinct algorithmic categories, and 95.6% of reviewed lineages introduce at least one category absent from their seed.Seed tasks are dominated by Search/simulation at 32.7% of seed-tag mentions, whereas evolved tasks cover advanced regimes including HLD/LCT, AC automata, and polynomial/matrix methods.
LIVECODEBENCH-PLUS: 91 problems: LIVECODEBENCH-PLUS combines 64 evolved problems—44 Hard and 20 Medium—with 27 difficult original LiveCodeBench-v6 problems.The evolved tasks preserve LiveCodeBench’s executable stdin–stdout interface after quality and difficulty filtering.
LIVECODEBENCH-PLUS: 41.3 points: average Hard-split pass@1 drops from 87.0% on source seeds to 45.7% on evolved tasks, while Medium drops from 96.5% to 69.6%, a 26.8-point reduction.The decrease holds for every individual model; GPT-5.4 falls from 94.8% to 49.7% on the Hard split.
Reinforcement learning: +3.6 points: on LCB v6 Hard, seed+evolved RL adds this gain over seed-only training; on LCB-Pro Easy, the corresponding additional gain is +2.1 points.Seed+evolved training performs best in both public held-out settings, suggesting evolved tasks complement the original seed distribution.

5 Conclusion and Future Work

BenchEvolver turns saturated executable coding tasks into harder, verified challenges by evolving reference solutions first and deriving statements and tests from the evolved computation. Future work focuses on scaling this self-improvement loop and maintaining reproducible, calibrated benchmark evolution as models improve.

Conclusion: BenchEvolver evolves reference solutions before recovering statements and tests, grounding task generation in executable semantics while producing valid, diverse, harder challenges.Across LiveCodeBench and SciCode, it substantially reduces target-model pass rates.
Scaling closed-loop RL self-improvement: A multi-round self-improvement loop could repeatedly use the improved model as the next evolver to generate new verified challenges and improve held-out coding performance.The current RL experiments instantiate one round of this loop using executable rewards.
Scaling closed-loop RL self-improvement: Scalable self-challenging generation requires controlling task difficulty, preserving algorithmic diversity, and balancing original, evolved, and newly evolved tasks.Narrow selection may overfit recurring failure modes, whereas aggressive selection may produce difficult but uninformative tasks.
Toward living benchmarks: Future benchmarks could use a reproducible evolution pipeline that periodically generates, validates, audits, and calibrates new tasks against current frontier models.This approach addresses eventual saturation of fixed benchmarks, particularly in executable domains where public tasks provide training signal.

A Additional Related Work … C.1 LiveCodeBench brute-force triangulation

BenchEvolver extends prior synthetic-data, self-play, and evolutionary-search ideas into solution-centric benchmark generation. Its procedure evolves executable coding tasks, validates them through consistency and difficulty checks, and applies brute-force triangulation for LiveCodeBench candidates.

A Additional Related Work: Synthetic coding-task work addresses the cost of collecting and curating human-written programming problems at scale.Prior approaches include evolving seed instructions and generating code-alignment instruction data.
A Additional Related Work: Self-play research generates model-targeted supervision through verifiable tool-use tasks or solver–conjecturer loops for theorem proving.These methods use model-in-the-loop generation to support evolving capabilities.
A Additional Related Work: BenchEvolver mutates an executable reference solution from a real coding task, then derives its statement and tests, requiring specification, executability, and empirical difficulty.This distinguishes complete evolved benchmark items from free-form questions generated from scratch.
A Additional Related Work: BenchEvolver redirects closed-loop evolutionary search from optimizing solutions, prompts, programs, or agents toward evolving harder executable benchmark objectives.The resulting tasks include reference solutions and tests, enabling self-improvement through a changing objective.
B Pseudocode for BenchEvolver: Each task is I = (S, C, T, E), while the algorithm tracks solver attempts, difficulty, accepted lineages, local histories, and globally accepted mutation ideas.The mutator, statement writer, and test generator operate within this task and history representation.
B Pseudocode for BenchEvolver: The evolution loop uses the latest accepted child as parent, evolves code first, derives statements and tests, validates executable consistency with bounded repair, and selects sufficiently harder non-artificial tasks.Rejected candidates record repair outcomes, failure patterns, and rejection reasons for subsequent evolution.
C.1 LiveCodeBench brute-force triangulation: For LiveCodeBench, validation synthesizes an independent brute-force solver and uses a three-way public-example vote among the reference solution, brute-force solver, and statement-only natural-language oracle.Candidates are rejected when disagreements cannot be resolved within a shared repair budget of three attempts.

C.2 SciCode statement-faithfulness validation

SciCode statement faithfulness is validated through a best-of-N procedure because its scientific, function-level tasks and assertion-based tests do not naturally support brute-force validation. An evaluator model solves each generated task from the statement alone, and alternate solutions are executed against generated tests to measure the best pass rate.

Validation method: SciCode tasks require best-of-N statement-faithfulness validation because their scientific function-level format, assertion-based tests, and domain-specific conventions hinder brute-force checking.The procedure evaluates whether the generated statement faithfully specifies the reference solution and hidden assertion tests.
Validation method: The evaluator model solves each generated task using only its statement, without access to the reference solution or hidden tests.This tests whether the statement alone provides sufficient information to reproduce the intended behavior.
Validation method: Each alternate solution is executed against the generated tests, and the highest pass rate across N attempts is retained.The retained best pass rate is the statement-faithfulness criterion used for SciCode validation.

D Training Details … D.3 Reward Design

The RL setup fine-tunes gpt-oss-20B with sandbox-grounded on-policy rewards, using LoRA, group-relative advantages, and a correctness-focused reward design. Training spans specified optimization settings, trajectory grouping, code extraction, test execution, and penalties for malformed or incomplete outputs.

D Training Details: The policy fine-tunes openai/gpt-oss-20B with on-policy RL through Tinker, using LoRA adapters and isolated cloud-sandbox execution for outcome-based rewards.The reference policy and sampler share base weights, while generated programs are evaluated against real test cases rather than a learned reward model.
D.1 Optimization and Model Configuration: The model uses rank r=32 LoRA adapters, learning rate η=1×10−5, one optimizer substep K=1, and no KL penalty βKL=0.The LoRA bottleneck and group-relative advantages help keep the policy close to the base model.
D.2 RL Objective and Batch Construction: Each task contributes G=16 trajectories, and B=64 task groups yield B · G=1024 trajectories per gradient step.Group-relative advantages center each trajectory’s reward against the mean reward for its own task group.
D.2 RL Objective and Batch Construction: Group-relative centering preserves gradient signal for both easy tasks with mostly passing rollouts and hard tasks with mostly failing rollouts.Training uses a clipped-importance-weight policy-gradient loss in the standard on-policy form used by Tinker’s RL trainer.
D.3 Reward Design: Rewards are computed after each conversation, by extracting the last fenced code block and submitting it with LiveCodeBench-format tests to the cloud sandbox.The grader evaluates the completed trajectory rather than issuing intermediate rewards.
D.3 Reward Design: A fully correct solution receives r=1, an incorrect formatted attempt receives r=0, and a response without extractable code receives r= −0.1.The small format penalty discourages chain-of-thought-only collapse while keeping format shaping subordinate to correctness.
D.3 Reward Design: Sandbox grading enforces a per-test wallclock limit of T=6 s, with stdin/stdout and functional problems evaluated through their respective harness paths.Output checks use exact match, numeric tolerance 10−6, and a token-set fallback; all test failures count as c=0.
D.3 Reward Design: Token-budget exhaustion before a final answer incurs roverflow= −0.1, preventing the policy from learning to stall indefinitely in the reasoning channel.The overflow penalty matches the format penalty in magnitude.

D.4 Training Dynamics

Training reward dynamics differ across RL data mixtures because seed tasks are easier for the base policy, whereas evolved tasks are intentionally harder and begin with substantially lower reward. Figure 7 tracks reward and average response length for seeds, evolved tasks, and their combination.

Training dynamics: Seed-task training starts with higher reward, while evolved-task training starts substantially lower because evolved problems are harder for the base policy.The initial reward gap reflects BenchEvolver’s selection of evolved tasks to expose failures of the current model.
Training dynamics: Figure 7 reports on-policy training reward and average response length for three mixtures: seeds, evolved tasks, and seeds plus evolved tasks.Curves show means from two random seeds per condition with shaded ±1 standard-deviation bands and width-5 boxcar smoothing.

D.5 Compute and Reproducibility · E Evolution Configurations and Hyperparameters

The experiments use a reproducible, service-based setup with fixed hyperparameters and two seeds per data mixture. Evolution configurations specify model settings, acceptance criteria, repair and validation budgets, test-generation parameters, and memory mechanisms.

D.5 Compute and Reproducibility: 40 hours and about $800 in Tinker credits are required for each RL run.All runs use Tinker for policy execution and gradient updates.
D.5 Compute and Reproducibility: Two independent random seeds are used for each data mixture, with hyperparameters fixed across all runs.This setup supports reproducibility across training runs.
D.5 Compute and Reproducibility: One CPU host runs the client-side training loop while Tinker performs policy execution and gradient updates.The division separates local orchestration from service-side training operations.
D.5 Compute and Reproducibility: Modal-hosted sandboxes execute code, making generated-program execution and test-case latency the primary throughput determinants.Local compute is not the main throughput bottleneck.
E Evolution Configurations and Hyperparameters: Table 5 summarizes default configurations for the LiveCodeBench and SciCode evolution experiments.These defaults define the standard setup used across both experimental domains.
E Evolution Configurations and Hyperparameters: The evolution pipeline specifies model settings and acceptance criteria as part of its default configuration.These parameters govern which generated candidates are evaluated and accepted.
E Evolution Configurations and Hyperparameters: Repair and validation budgets are included among the pipeline’s default evolution hyperparameters.The configuration explicitly allocates resources for repairing and validating evolved solutions.
E Evolution Configurations and Hyperparameters: Test-generation parameters and memory mechanisms complete the reported default evolution configurations.Table 5 records both test-generation behavior and memory-related components.

F Human Evaluation: Full Breakdown

A blinded expert review found that BenchEvolver’s evolved problems were substantially harder and more novel than their seeds, while remaining well specified and algorithmically diverse. The evaluation used six competitive-programming experts and anonymized review groups to reduce bias.

Review protocol: Six competitive-programming experts blindly reviewed anonymized groups of 2 to 4 problems, without knowing seeds, evolved variants, or generating models.Reviewers rated clarity, novelty, and difficulty and certified that they did not use generative AI during evaluation.
Human-evaluation results: 1.83 → 3.21: evolved problems received substantially higher difficulty ratings than seed problems.Estimated Codeforces ratings also shifted from a concentrated range around 1100 to a broader distribution centered near 2100.
Human-evaluation results: 2.21 → 3.10: evolved problems were judged more novel than seeds, with fewer close-variant ratings.The evaluation reported a consistent shift from seed problems to evolved problems.
Human-evaluation results: 95.6% of lineages introduced at least one new algorithmic category, averaging 2.54 new categories per group.The total number of observed categories increased from 19 to 30, supporting diversity while problems remained well specified.

G Examples of Evolved Benchmark Items … I Prompt Templates for BenchEvolver

BenchEvolver’s examples show how solution-centric mutations create harder LiveCodeBench and SciCode tasks by expanding the algorithmic scope while preserving a clear semantic target. The appendices also illustrate evolution trajectories and prompt templates that coordinate solution mutation, statement generation, and validated test construction.

G Examples of Evolved Benchmark Items: BenchEvolver compares successfully evolved frontier tasks for LiveCodeBench and SciCode against their seed problems.The examples are organized around solution mutations and their resulting algorithmic lifts.
LCB Seed: Copy Arrays → Evolved Problem: XOR-Linked Sequence: The XOR-linked sequence replaces one additive offset and contiguous interval intersection with noncontiguous XOR constraints requiring bitwise digit-DP or trie-style methods.The seed is solvable by interval intersection in O(N), whereas the evolved constraints take the form ui ≤ x ⊕ pi ≤ vi.
Evolved #1: Maximum Unique Value Query / Evolved #2: Maximum Unique Path Value: Range and path variants replace one global frequency pass with range-dependent frequency profiles, offline or data-structural reasoning, path decomposition, and tree-topology reasoning.The evolved settings shift uniqueness evaluation from whole-array scope to contiguous subarrays and then tree paths.
LCB Seed: Uniqueness → Evolved #2: Maximum Unique Path Value: The uniqueness task progressively generalizes from selecting a globally unique maximum, to range-query uniqueness, to path-query uniqueness on a tree.The semantic target remains identifying the largest value unique within the relevant scope and returning its index.
SciCode Seed: 4th-Order Runge-Kutta Integrator / EXAMPLE TEST: The SciCode RK4 seed performs fixed-ODE forward simulation, while the evolved ODE task estimates unknown initial states and parameters from sparse observations using damped Gauss-Newton, finite-difference Jacobians, and backtracking line search.This turns direct numerical integration into a nonlinear parameter-estimation pipeline while retaining RK4 trajectory simulation between observation times.
EXAMPLE TESTS / Evolved: Maximum Pairwise CHSH Bell Value: The n-tangle seed computes one global entanglement scalar for an even-qubit pure state, whereas the evolved task searches all two-qubit subsystems for the largest attainable CHSH Bell value.The mutation changes a single functional evaluation into comparing pairwise nonlocal correlations across candidate subsystems.
H Examples of Evolution Trajectory: Figure 10 presents representative BenchEvolver trajectories in which each row tracks a seed problem across accepted evolution rounds using pooled passes over attempts and solve-rate coloring.The appendix describes Gemini-3-Flash trajectories on the LiveCodeBench Easy split.
I Prompt Templates for BenchEvolver: BenchEvolver’s prompt templates assign distinct roles to solution mutation, statement writing, and test generation, emphasizing a single genuine algorithmic lift, code-authoritative semantics, explicit constraints and tie-breaking, and deterministic tiered validation.The templates use evolution memory and target-model weaknesses, require valid JSON, and make validate() the sole source of input legality before reference outputs become ground truth.

J Reproducibility and Asset Licenses

The authors plan to release the evolution pipeline code, configurations, and experimental assets, while documenting benchmark provenance and complying with applicable licenses. Released generated artifacts will include attribution and license information.

Code and configurations: The released repository will include evolution-pipeline code, configurations, scripts, and default experimental hyperparameters.Scripts cover seed selection, candidate generation, validation, target-model evaluation, and result aggregation.
Benchmarks and datasets: The experiments use LiveCodeBench Version 6 by seed difficulty and selected SciCode validation subproblems under research-use terms.The paper cites the original benchmark papers and repositories and follows their corresponding licenses and terms of use.
Licenses: LiveCodeBench is used under the MIT License, while SciCode is used under Apache License 2.0.LiveCodeBench data comes from its official repository, and the SciCode dataset comes from Hugging Face under Apache-2.0.