Source-linked AI summary
FrontierCS: Evolving Challenges for Evolving Intelligence
Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, Xieting Chu, Zerui Cheng, Yuan Xu, Tian Xia, Zirui Wang, Tianneng Shi, Jianzhu Yao, Yilong Zhao, Qizheng Zhang, Charlie Ruan, Zeyu Shen, Kaiyuan Liu, Runyuan He, Dong Xing, Zerui Li, Zirong Zeng, Yige Jiang, Lufeng Cheng, Ziyi Zhao, Youran Sun, Wesley Zheng, Meiyuwang Zhang, Ruyi Ji, Xuechang Tu, Zihan Zheng, Zexing Chen, Kangyang Zhou, Zhaozi Wang, Jingbang Chen, Aleksandra Korolova, Peter Henderson, Pramod Viswanath, Vijay Ganesh, Saining Xie, Zhuang Liu, Dawn Song, Sewon Min, Ion Stoica, Joseph E. Gonzalez, Jingbo Shang, Alvin Cheung
TL;DR
FrontierCS addresses the lack of a comprehensive benchmark for open-ended computer-science problems whose optima are unknown but whose solutions can still be objectively scored. It constructs expert-curated algorithmic and research tasks requiring executable programs and deterministic evaluation, and finds that frontier reasoning models remain far behind human experts. The benchmark also shows diminishing returns from larger reasoning budgets and supports evolving task difficulty while preserving comparability.
Problem
Existing code and reasoning benchmarks largely use closed-form tasks, leaving limited comprehensive evidence about model performance on open-ended, cross-domain CS problems.
Method
FrontierCS provides 156 expert-curated open-ended problems across algorithmic and research tracks, evaluated through executable submissions, automatic scoring, and reproducible environments.
Results
Frontier reasoning models remain far behind human experts on both tracks, and increasing context length or reasoning budgets yields diminishing returns on the hardest problems.
Takeaways & Limitations
FrontierCS provides a systematic testbed for measuring progress on open-ended CS reasoning and for evolving benchmark difficulty as models improve.
Takeaways & Limitations
Evaluation is single-round and text-only, with no code execution, unit-test feedback, iteration, editor, Python environment, or external tools.
Abstract
from arXiv · showhide
We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts on both the algorithmic and research tracks, that increasing reasoning budgets alone does not close this gap, and that models often over-optimize for generating merely workable code instead of discovering high-quality algorithms and system designs.
1. Introduction
FrontierCS addresses the lack of a comprehensive cross-domain benchmark for open-ended computer-science problems by evaluating executable solutions whose quality can be objectively scored. It spans algorithmic and research tasks without known practical optima, and finds that frontier reasoning models remain far behind experts.
- Existing code and reasoning benchmarks largely use closed-form, single-answer, pass-or-fail tasks, leaving no comprehensive cross-domain testbed for open-ended CS problems.Recent open-ended efforts are typically limited to particular application domains or small sets of representative cases.
- FrontierCS evaluates executable programs on open-ended problems whose unknown optima are replaced by deterministic validity checks and quantitative quality scores.Its tasks include expert reference solutions and automatic evaluators, enabling objective comparisons and reproducibility.
- Open-ended tasks such as Polyomino Packing admit many valid solutions with continuously varying quality, measured through metrics such as packing density.The evaluator checks validity before assigning a quality score.
- The benchmark covers algorithmic optimization and real-world research tasks, encouraging iterative improvement rather than pursuit of unavailable practical optima.Objective feedback also makes the benchmark suitable for evaluation, training, and ablation studies.
- Even the strongest frontier reasoning models remain far behind human experts, while larger context or reasoning budgets yield diminishing returns on the hardest problems.Models often settle for locally workable but clearly suboptimal algorithms.
2. Related Work
Prior benchmarks evaluate binary correctness, runtime or performance, or subjective judgments, but generally cover narrower task types or depend on limited authorship. FrontierCS combines diverse open-ended problems with deterministic, continuously scored evaluation.
- FrontierCS contains 156 problems across algorithmic and research tracks, with research tasks spanning six computer-science domains.The algorithmic track includes optimization, constructive, and interactive categories.
- Earlier open-ended benchmarks such as ALE-Bench cover narrower problem slices and draw on a small number of recurring problem setters, limiting diversity.Other benchmarks include unsolved forum tasks or GPU-kernel optimization, but target more specific settings.
- FrontierCS differs from existing paradigms by combining unsolved, automatically verifiable problems with continuous solution-quality scores instead of binary correctness or human judgment.This supports partial progress on research-style tasks whose optima are unknown or computationally prohibitive.
3. Problem Collection
FrontierCS builds a 156-problem collection through expert proposal, implementation, and review, with deterministic partial scoring and reproducible execution across algorithmic and research tracks. Its update policy increases difficulty while preserving task continuity and comparability.
- 3. Problem Collection: The collection contains 107 algorithmic problems and 49 research problems, covering contest-derived categories and six real-world CS research domains.The algorithmic track includes 29 optimization, 27 constructive, and 51 interactive problems.
- 3.1. Algorithmic Problems: Algorithmic tasks are curated through expert proposal, implementation, and review to ensure open-ended solutions, meaningful objective scoring, valid evaluation, and stronger human references.Implementation adds generators, deterministic verification, partial scoring, reference solutions, baselines, and reproducible harnesses.
- 3.1. Algorithmic Problems: Problem-specific metrics such as cost, density, or query count primarily determine scores, while strict time and memory limits reject invalid solutions without making runtime itself the objective.This separates strategy quality from excessive computation.
- 3.2. Research Problems: Research tasks are based on unsolved questions and use specialized reproducible environments with pinned dependencies, exact input/output contracts, and deterministic evaluate.sh scoring.Accepted problems include expert references, baselines, and diagnostics; evaluators cannot use an LLM as judge.
- 3.3. Update Policy: FrontierCS can evolve by adding tasks or increasing existing-task difficulty through tighter constraints, changed workloads, or stricter objectives without changing problem statements.These mechanisms preserve task continuity and comparability across versions.
4. Evaluation Results
FrontierCS evaluates frontier models against human experts using objective, partially scored solutions across algorithmic and research problems. Models benefit from multiple trials, but remain substantially behind expert performance.
- Evaluation uses Score@1, Avg@5, Score@5, Pass@1, and Pass@5 to capture both solution quality and stochastic gains across repeated trials.
- Human experts achieve 95.41 Score@1 on algorithmic problems, versus 5.84–29.37 across the eight reported frontier models.
- Score@5 improves over Score@1 by 6.40–22.69 points across algorithmic models, showing that additional sampling produces stronger candidate solutions.
- On research problems, Claude Opus 4.5 leads one-shot performance at Score@1 29.40, while GPT 5.1 Thinking leads multi-sample performance at Score@5 47.21.
- Research-track Score@5 improves over Score@1 by 6.72–21.04 points, but pass rates remain substantially higher than raw scores, indicating many workable yet under-optimized solutions.
5. Discussion
Discussion analyses why increased reasoning effort does not reliably improve open-ended performance and identifies a recurring tendency toward workable but strategically weak solutions. Targeted structural guidance can substantially improve outcomes in specific tasks, while engineering competence and optimization remain distinct challenges.
- 5.1. Improving Reasoning Effort Does Not Yield Further Gains: Increasing GPT 5 Thinking’s effort from medium to high reduces performance from 15.336 to 12.626, indicating diminishing returns beyond an upper reasoning-budget limit.
- 5.2. Misleading Micro-Optimization Trap: In Polyomino Packing, transformation-list representations make overlap detection and free-space search cumbersome, producing invalid code in about 30% of attempts and scores of 20–70 otherwise.
- 5.2. Misleading Micro-Optimization Trap: Adding one instruction to maintain a 2D array and convert formats only at the end drops the zero-score rate to about 10%, with nearly 80% of cases reaching scores of 80–85.
- 5.3. The Research–Engineering Dilemma of Claude: Claude Sonnet 4.5 often produces workable outputs without competitive optimization, whereas broader research tasks reward both engineering skills and effective optimization.
6. Example Problems
FrontierCS examples span constructive, optimization, interactive, research, security, and systems problems where valid solutions receive graded scores despite unknown optima. Across these tasks, human solutions substantially outperform model-generated approaches in solution quality, compactness, packing, and strategic performance.
- Example Problems: Constructive examples impose global structural constraints, such as adjacency requirements for labeled grids and non-overlapping placement of unit squares.The square-packing objective minimizes the container side length, with scores based on validity and closeness to lower-bound and reference solutions.
- Example Problems: FrontierCS examples use objective grading for open-ended problems whose exact optima may be unknown or infeasible to compute.Approximate solutions are compared using task-specific objectives such as value, density, grid size, query count, or latency while validity and resource limits remain enforced.
- Example Problems: Human experts outperform models on representative constructive and optimization tasks, including grid construction, treasure packing, square packing, and polyomino packing.The treasure-packing model scores 74 versus the human score of 100, while other examples report larger model grids or more whitespace than human solutions.
- Example Problems: The examples include interactive tasks that require identifying a hidden permutation through adaptive queries.The permutation-guess problem uses n = 1000 and asks models to minimize the number of queries needed to identify the hidden permutation.
- Example Problems: Research-track examples extend the benchmark to symbolic regression, approximate-nearest-neighbor systems, cybersecurity fuzzing, and kernel optimization.Reported comparisons include human symbolic-regression complexity 12 versus GPT 5 complexity 19 and human versus GPT 5 PoC sizes of 79 versus 577 bytes.
- Example Problems: Strategic planning examples show models below a simple human strategy, with GPT 5 and Gemini 2.5 Pro scoring 25 and 36 versus 54 for the human approach.The human strategy uses Monte Carlo estimates and always goes all-in when the winning probability exceeds 0.75.
7. Conclusion
FrontierCS is a comprehensive benchmark for open-ended computer-science tasks whose unknown optima remain deterministically verifiable and partially gradable. Its initial study finds current LLMs brittle on open-ended optimization and system-level trade-offs, with closed-form coding competence not reliably transferring.
- FrontierCS provides a diverse, systematic testbed for tasks with unknown global optima but deterministic verification and partial scoring.The benchmark includes expert reference solutions, an automated evaluator, a reproducible pipeline, and versioned difficulty scheduling.
- Its initial study shows current LLMs remain brittle on open-ended optimization and system-level trade-offs.
- Competence on closed-form coding tasks does not reliably translate into open-ended computer-science problem solving.