Source-linked AI summary
FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale
Runyuan He, Qiuyang Mang, Shang Zhou, Kaiyuan Liu, Hanchen Li, Huanzhi Mao, Qizheng Zhang, Zerui Li, Bo Peng, Lufeng Cheng, Tianfu Fu, Yichuan Wang, Wenhao Chai, Jingbo Shang, Alex Dimakis, Joseph E. Gonzalez, Alvin Cheung
TL;DR
Open-ended coding lacks abundant, inexpensive training problems because its tasks require continuous evaluation and often have no tractable optimality certificate. FrontierSmith evolves closed-ended seeds into open-ended variants, filters them using idea divergence, and generates test cases and verifiers; training on the resulting data improves two open-ended benchmarks and produces long-horizon agent behavior similar to human-curated tasks. The pipeline is currently limited to self-contained algorithmic environments and single-turn GRPO training.
Problem
Open-ended coding lacks an equivalent to the large, verified data sources available for closed-ended tasks, while its problems are scarce and expensive to construct.
Method
FrontierSmith iteratively mutates closed-ended competitive-programming problems along three axes, selects candidates with idea-divergence filtering, and generates test cases and verifiers.
Results
+8.82 score on FrontierCS and +306.36 Elo-rating-based performance on ALE-bench are gains for Qwen3.5-9B trained on synthesized problems, with similar gains for Qwen3.5-27B.
Takeaways & Limitations
Closed-ended seeds can support scalable synthesis of open-ended coding data whose training gains and agent behavior are comparable to those associated with human-curated problems.
Takeaways & Limitations
The pipeline supports only self-contained algorithmic environments and the training study uses 100 steps of single-turn GRPO, leaving repo-level tasks and agentic RL unexplored.
Abstract
from arXiv · showhide
Many real-world coding challenges are open-ended and admit no known optimal solution. Yet, recent progress in LLM coding has focused on well-defined tasks such as feature implementation, bug fixing, and competitive programming. Open-ended coding remains a weak spot for LLMs, largely because open-ended training problems are scarce and expensive to construct. Our goal is to synthesize open-ended coding problems at scale to train stronger LLM coders. We introduce FrontierSmith, an automated system for iteratively evolving open-ended problems from existing closed-ended coding tasks. Starting from competitive programming problems, FrontierSmith generates candidate open-ended variants by changing the problems'goals, restricting outputs, and generalizing inputs. It then uses a quantitative idea divergence metric to select problems that elicit genuinely diverse approaches from different solvers. Agents then generate test cases and verifiers for the surviving candidates. On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models: Qwen3.5-9B improves by +8.82 score on FrontierCS and +306.36 (Elo-rating-based performance) on ALE-bench; Qwen3.5-27B improves by +12.12 and +309.12, respectively. The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones, suggesting that closed-ended seeds can be a practical starting point for long-horizon coding data.
1 Introduction
FrontierSmith addresses the scarcity and cost of open-ended coding data by evolving closed-ended problems into open-ended variants and filtering them for diverse solution strategies. Training on the resulting problems improves performance across two open-ended benchmarks and elicits long-horizon agent behavior similar to human-curated tasks.
- Motivation: Open-ended coding lacks the abundant, efficiently verifiable training data available for closed-ended tasks, while manually constructing problems requires objectives, continuous-score verifiers, test cases, and expert judgment.FrontierCS and ALE-bench contain only around 240 and 40 human-curated problems, respectively.
- Approach: Idea divergence selects candidates whose sampled solutions express genuinely different core strategies, while filtering and automated test-case and verifier generation validate the surviving problems.The system uses filtering to retain problems with optimization objectives, multiple plausible strategies, and meaningful scoring functions.
- Approach: FrontierSmith evolves closed-ended competitive-programming seeds into open-ended variants by changing goals, restricting outputs, and generalizing inputs.These mutations transform problems with known efficient solutions into tasks with intractable exact solutions and diverse strategies.
- Results: +8.82 score on FrontierCS and +306.36 Elo-rating-based performance on ALE-bench are gains for Qwen3.5-9B trained on 200 synthesized problems.Qwen3.5-27B gains +12.12 on FrontierCS and +309.12 on ALE-bench.
- Results: FrontierSmith-generated problems yield training gains comparable to human-curated data and substantially outperform closed-ended training across benchmarks.The reported comparison includes gains over closed-ended HardTests and random-reward controls.
- Results: Synthetic problems elicit more turns, tool calls, and thinking tokens from agents, matching long-horizon behavior observed on human-curated open-ended tasks.The authors interpret this behavioral match as evidence that the synthetic problems capture similar structure.
2 Related Work
Prior coding-data synthesis largely targets closed-ended binary-correctness settings, while open-ended benchmarks evaluate continuous solution quality. Related work also includes hand-crafted open-ended evaluation and mutation-based evolutionary methods, but does not provide the same open-ended problem-synthesis source.
- Coding data synthesis for closed tasks: Existing coding-data synthesis methods generate problems and test infrastructure primarily for closed-ended tasks evaluated by binary correctness.The surveyed methods span competitive programming, software engineering, terminal environments, and self-play.
- Open-ended evaluation and solution diversity: Open-ended coding benchmarks score solutions on continuous quality scales, and prior systems have hand-crafted instance generators and rule-based verifiers for selected optimization tasks.The related work lists FrontierCS, ALE-bench, HeuriGym, KernelBench, RE-Bench, and MLE-bench.
- Mutation-based synthesis and iterative self-play: Mutation-based methods evolve prompts or programs, while iterative bootstrap methods recycle self-generated data through training; these approaches do not themselves supply open-ended problem formulations.The cited evolutionary work operates on prompts or programs rather than a scalable source of open-ended coding problems.
3 Method
FrontierSmith evolves closed-ended coding problems into open-ended candidates, filters them for optimization quality and strategy diversity, then synthesizes validated test cases and verifiers. Its pipeline combines formulation mutations, idea-divergence estimates, and cross-validated testing infrastructure to retain usable candidates.
- Mutation Problem Formulation: FrontierSmith starts from closed-ended coding tasks and mutates goals, output constraints, or input assumptions to create open-ended variants.These mutations aim to remove efficiently certifiable optima while preserving continuous quality evaluation.
- Mutation Problem Formulation: The system represents a problem as (O, CI, CO), respectively its computational goal, admissible instances, and valid-output constraints.Mutation acts on one or more components of this formulation.
- Problem Filtering: Candidates pass a coarse judge that checks for an unknown optimum, multiple plausible strategies, and a meaningful scoring function.Candidates failing any condition are discarded before diversity-based selection.
- Problem Filtering: Idea divergence estimates the probability that two independently generated solutions use different algorithmic strategies.The pipeline first uses LLM judgments of sampled solution pairs, then uses execution-based score behavior after test infrastructure exists.
- Testing Infrastructure: Surviving candidates receive generated test inputs and a verifier that scores solutions continuously, with sampled solutions used to cross-validate both agents’ outputs.The test-case agent targets varied structures and strategy failures, while the verifier translates the objective into normalized scores.
- Testing Infrastructure: 10% of candidates entering testing infrastructure produce a validated test-case and verifier pair; nonconvergent candidates are discarded.The validated pair feeds execution-based idea-divergence reranking.
4 Experiments
FrontierSmith is evaluated through RL training on two open-ended coding benchmarks, with comparisons against human-curated, closed-ended, random-reward, and filtering controls. Its synthesized data performs competitively or better across model sizes, while divergence filtering and source analysis support open-endedness and longer agent horizons.
- Experimental Setup: The evaluation uses FrontierCS’s 172 algorithmic problems and 40 ALE-bench tasks, with five training configurations including HardTests and Random Reward.Table 1 reports best checkpoints after 100 training steps, using Avg@5 and Best@5 metrics.
- Main Results: FrontierSmith achieves competitive or superior performance to human-curated training data on both benchmarks across Qwen3.5-9B and Qwen3.5-27B.The reported results indicate that performance gains hold across model sizes.
- Main Results: 10.62 Avg@5 on FrontierCS is FrontierSmith’s Qwen3.5-9B result, close to 11.17 from human-curated FrontierCS training.On ALE-bench, FrontierSmith reaches the highest Best@5, 782.30, and an Avg@5 of 633.58.
- Main Results: 19.82 versus 13.98 on FrontierCS and 661.64 versus 543.80 on ALE-bench: FrontierSmith outperforms human-curated training on Qwen3.5-27B.The corresponding base-model scores are 7.70 and 352.52.
- Main Results: 10.62 versus 8.57 on FrontierCS and 633.6 versus 564.4 on ALE-bench: filtered FrontierSmith data outperforms the no-filter variant.Both filtered and unfiltered synthetic data outperform HardTests, whose scores are 5.38 and 397.2.
- Filter Analysis: 91 of 100 closed-ended problems are rejected by the coarse filter, producing a 9% false-positive rate; applying it to FrontierCS yields a 19% false-negative rate.The filter prioritizes preserving high-quality open-ended problems rather than maximizing recall.
- Filter Analysis: 0.42 versus 0.40: FrontierSmith’s LLM-based idea-divergence score slightly exceeds human-curated FrontierCS and remains roughly 3× HardTests’ 0.14.The execution-grounded estimate also separates open-ended sources from HardTests.
- Long-Horizon Code Agent Behavior: 113 turns and 6.3×10^6 tokens: FrontierSmith elicits long-horizon behavior from Claude SDK, comparable to ALE-bench and unlike HardTests or FrontierCS.The analysis uses geometric-mean turns and tokens across source-agent pairs.
5 Discussion
FrontierSmith currently targets self-contained algorithmic environments and uses limited single-turn GRPO training. Extending it to repo-level tasks and agentic RL remains future work.
- Limitations: The pipeline excludes repo-level tasks requiring complex environment setup, including cloud optimization, GPU kernel tuning, and multi-file software engineering.Such settings would require reproducible execution environments in addition to problem formulations and verifiers.
- Limitations: RL training is limited to 100 steps of single-turn GRPO, and agentic RL is not explored because of compute constraints.The paper identifies multi-turn environment interaction as a future direction aligned with synthesized problems’ long-horizon nature.
6 Conclusion
FrontierSmith transforms closed-ended coding problems into open-ended ones through targeted mutations and idea-divergence filtering. Its synthesized problems support competitive training performance and long-horizon agent behavior without expensive expert curation.
- FrontierSmith transforms closed-ended coding problems into open-ended ones using targeted mutations and idea-divergence filtering.
- Training Qwen3.5-9B and Qwen3.5-27B on FrontierSmith-generated problems achieves performance competitive with training on human-curated open-ended data.
- Synthesized problems elicit long-horizon agent behavior comparable to human-curated tasks, with agents spending substantially more turns and tokens than on closed-ended problems.
- FrontierSmith provides a scalable source of training data for reinforcement learning on long-horizon, open-ended coding tasks.
A Example Synthesized Problems
FrontierSmith produces example open-ended problems by mutating closed-ended seed problems. The examples shown come from different seeds.
- The section presents two example open-ended problems produced by FrontierSmith.
- Each example is mutated from a closed-ended seed.
- The two examples originate from different closed-ended seeds.
A.1 Concat Factory Compression Challenge
The Concat Factory Compression Challenge uses a toy language with literal creation and concatenation commands. Solvers must construct all target strings in a valid, minimal program.
- Polycarp’s toy language contains literal creation and concatenation commands.Literal creation makes lowercase strings, while concatenation combines previously created variables.
- Literal creation creates a variable containing a lowercase string of length 1–8.
- Concatenation creates a variable by joining the values of two previously created variables.
- The task is to output a valid program that constructs every target string while keeping the program as small as possible.
Input
The input consists of multiple test cases, each specifying a number of target strings. Every target is provided on its own line.
- The first input line contains T, the number of test cases.
- Each test case begins with q, the number of targets.
- Each target string appears on a separate line after q.
Output
Solutions output a compact program and identify variables matching every target, while satisfying syntax, dependency, length, and budget constraints. Performance is evaluated against a reuse-free baseline using a capped ratio score.
- Output: Each test case outputs m commands followed by q variable indices whose values equal the target strings.Commands are either literals or concatenations of previously created variables.
- Validity: A feasible solution must obey command, literal-length, variable-reference, value-length, and target-matching constraints.The command count is bounded by 5000, literals use lowercase strings of length 1–8, and concatenations reference earlier variables.
- Scoring: The baseline cost splits every target into blocks of length at most 8 without reuse.The objective compares the submitted program’s cost with this baseline.
- Scoring: The test-case score is S = 10^6 · min(2, B/C), and the final score is the mean across test cases.Here B is the baseline cost and C is the feasible solution cost.
- Road-patrol output: The road-patrol task selects roads and daily patrols to minimize damage from attacks while respecting construction, movement, and return-to-base constraints.Each patrol starts and ends at house 1, uses built roads, and must satisfy its daily time limit.
- Road-patrol scoring: Its output lists built road indices and one valid closed walk for each day, with score based on the baseline-to-submitted damage ratio.The score is capped at 10×10^6 and averaged over test files.