PaperlayerView exact revision on arXiv

Source-linked AI summary

FrontierCS 2: A Benchmark for Computer Science R&D

Qiuyang Mang, Wenhao Chai, Runyuan He, Dongruixuan Li, Zihan Zheng, Yujun He, Xuanyi Zhou, Hao Wang, Siddhant Ray, Bo Peng, Ruyi Ji, Shuo Yang, Haocheng Xi, Boru Chen, Xinyang David Han, Melissa Pan, Mayank Mishra, Boxi Yu, Sijia Liu, Kerem Akillioglu, Yi Pan, Zirui Wang, Mert Cemri, Diogo Antunes, Yichuan Wang, Aiwei Ivy Zhang, Zhenyang Chen, Qizheng Zhang, Kaiyuan Liu, Wenyuan Huang, Sherry Ruan, Shu Liu, Dacheng Li, Zerui Cheng, Zeyu Shen, Shang Zhou, Hanfei Chen, Yuzhong Zhang, Bohan Lyu, Qu Ao, Han Zheng, Chonghe Jiang, Jingzhuo Hu, Xiaokun Chen, Kunjun Li, Zhiqing Zhong, Haoye Tian, Tong Zhao, Yunzhong He, Hanchen Li, Ofir Press, Jiantao Jiao, Kunle Olukotun, Karthik R. Narasimhan, Jingbo Shang, Koushik Sen, Matei Zaharia, Alex Dimakis, Ion Stoica, Joseph E. Gonzalez, Alvin Cheung

arXiv:2610.29932v1cs.LG

TL;DR

FrontierCS 2 asks whether LLM agents can improve published computer-science systems in their own repositories against the authors’ implementations. It builds patch-based tasks by removing each paper’s contribution and evaluating agents against that code, with Claude Opus 5.5 passing 58% of runs and no agent passing on 16 tasks.

  • Problem

    The paper asks whether an LLM agent can improve a published system in its working repository against the paper’s own implementation.

    Source: §1 · p. 3

  • Method

    The benchmark deletes a paper’s contribution from its repository, uses the removed code as the human reference, and evaluates an agent’s final patch on a held-out suite.

    Source: §1 · pp. 3–4, §3 · p. 7

  • Results

    Claude Opus 5.5 in Claude Code passes 58% of its runs, and no agent passes any of 16 tasks.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The pass and final score can miss image quality and side-cost distinctions, so benchmarks like this should report side costs alongside the pass.

    Source: §7 · p. 15

    The pass and final score can miss image quality, bytes removed per lookup, and circuit size, motivating separate side-cost reporting.

    Source: §7 · p. 15

Abstract

from arXiv · show

Can LLM agents make contributions to frontier computer science R&D? We introduce FrontierCS 2, a benchmark of 48 open-ended tasks. We take a paper and the code its authors released and delete the paper's contribution. The deleted code becomes the human reference that sets the bar. Earlier open-ended benchmarks give each agent a problem statement and grade one program after the run. FrontierCS 2 places the agent inside a research environment. The agent inherits a working repository and may submit a patch at any point. For each patch, a judge returns a development-suite score but keeps the evaluator, the held-out final suite and the reference private, and grades the final submission on that suite against the reference. The tasks span 13 domains, from machine-learning systems and high-performance computing to databases, security and robotics. Beyond these 48 tasks, FrontierCS 2 keeps a private set of more than 100 tasks, to be announced in the future. We evaluate 12 agents. The best one is Claude Opus 5.5, passing 58% of its runs. No run passes on 16 tasks. A higher score is not the same as a pass: most runs that beat the reference on the combined final score still fall short on some metric or workload. On a shared capability scale for agents and tasks, the median task is roughly as difficult as what the strongest agent can handle.

1 Introduction

FrontierCS 2 asks whether LLM agents can improve published computer-science systems in their own repositories against the authors’ code. Claude Opus 5.5 passes 58% of runs, while no agent passes on 16 tasks.

1 Introduction

  • FrontierCS 2 evaluates agents inside the repositories their papers came from, with the authors’ published code as the benchmark bar.Agents submit patches during a run; the judge keeps the evaluator, held-out final suite, and human reference private.Source: §1 · p. 3
  • Claude Opus 5.5 in Claude Code passes 58% of its runs, every other agent passes fewer than half, and no agent passes any of 16 tasks.The median task sits at about the strongest agent’s FCI.Source: §1 · p. 4
  • Tasks are built by deleting a paper’s contribution from its repository and retaining the deleted code as the human reference.The benchmark also retains a private set of more than 100 tasks for future announcement.Source: §1 · p. 4
  • FrontierCS 2 covers 48 tasks in 13 domains and evaluates 12 agents.The introduction also reports three case studies.Source: §1 · p. 4

2 Related work

Prior benchmarks vary in whether they use one-shot coding tasks or research environments, and in how they evaluate agent work. FrontierCS 2 distinguishes itself by using the paper authors’ code as the reference across 13 computer-science domains.

Open-ended coding benchmarks

  • Open-ended coding benchmarks score programs on objectives with no known optimum, whereas speed-up benchmarks use fixed baselines; FrontierCS 2 grades most tasks against authors’ released human reference and scores each patch during the run.Source: §2 · p. 6

Agent benchmarks for research and engineering

  • Research and engineering benchmarks use tests, leaderboards, rubrics, or published results, but none sets its bar by measuring the original authors’ code.MLS-Bench [7] is closest in design, with 140 machine-learning tasks across 12 areas that ask agents to improve a named component.Source: §2 · p. 6

Building tasks from existing artifacts

  • Artifact-based task-building methods start from working artifacts and break them in controlled ways; FrontierCS 2 uses the removed paper method itself as the oracle and spans 13 domains.Unlike ResearchGym [6], its oracle is the removed code.Source: §2 · p. 6

3 Task building

FrontierCS 2 builds tasks by removing a paper’s contribution from its released repository while leaving dependencies intact; the deleted code becomes the human reference. Agents restore the missing code under metric thresholds and held-out evaluation.

3 Task building

  • Deleting a paper’s contribution while leaving its dependencies intact creates a repository task with the deleted code as its human reference.The reference is a private patch that restores the deleted code and already runs on the paper’s workloads.Source: §3 · p. 7
  • For SVG-EAR, agents must restore the deleted attention routines and may edit only the files that held the contribution.The deleted routines average clusters, choose cluster pairs to compute, and compute those pairs while estimating the rest.Source: §3 · p. 7
  • Pass thresholds are set from the reference’s measurements, and the final suite adds held-out workload instances unseen by the agent.For SVG-EAR, latency must be at most 0.96 times the reference; PSNR, SSIM, and LPIPS must stay within specified margins.Source: §3 · p. 7
  • The agent sees development-suite scores and sanitized messages, while information used to decide its result stays with the judge.Structured LWE is an exception to the task-building approach, as it is an original cryptanalysis challenge.Source: §3 · p. 7, Figure 4 · p. 8

4 Judge

A separate judge scores patches during the run but keeps the evaluator, held-out final suite, and human reference private. Only the agent’s chosen final patch is graded against the reference on held-out workloads.

4 Judge

  • The agent has five hours to submit patches, while the judge returns development-suite scores and threshold counts without revealing the reference or raw evaluator output.The agent and judge run in separate sandboxes; the agent’s sandbox reaches the judge only through a driver.Source: §4 · p. 8
  • Only the agent’s chosen final patch is scored on held-out workloads; a pass requires beating the reference on every metric across every held-out workload.Development-suite scores guide the agent but do not count toward its result.Source: §4 · p. 8
  • KV Compaction and Progent define task-specific final scores from workload-level measures and report 100 times their mean.KV Compaction combines answer accuracy and speed, with a cache-size discount; Progent combines utility under attack with the share of unbreached episodes.Source: §4 · p. 8
  • GPT-6.1 Sol passes less often than GPT-6 Astra while sitting far lower on mean cost per run in Figure 5.The figure plots pass rate against mean run cost in US dollars on a log-scale axis.Source: Figure 5 · p. 9

5 Evaluation results

Across the evaluation, task difficulty spans a wider range than agent capability, and which agent leads depends on the per-run cost budget. Final scores and pass outcomes are not interchangeable because a pass requires meeting every metric on every held-out workload.

5.1 Experimental setup

  • The evaluation uses 12 model-harness agents, each run once on each task; two refused Gemini 4 Argon runs count as failures.The harness is part of what is measured.Source: §5.1 · p. 9
  • Most runs scoring above the human reference still miss a threshold on at least one metric or workload, so a higher final score does not ensure a pass.Some passing runs score at or below the reference because passing requires every metric to hold at once.Source: §5.1 · p. 9

5.2 FrontierCS Capabilities Index (FCI)

  • FCI and FDI come from a joint fit to per-metric outcomes and final-score margins over the human reference, with each task’s FDI marking a 50% pass point.The fit uses two observations per run: which metrics it beats the reference on and its final-score margin.Source: §5.2 · p. 10
  • The median task sits at about the strongest agent’s FCI, while task difficulty spans a wider range than agent strengths.Claude Opus 5.5 leads FCI, followed by GPT-6 Astra and GPT-6.1 Sol; individual tasks are not ordered because their FDI estimates are uncertain.Source: §5.2 · p. 10, Figure 7 · p. 11

5.3 Test-time scaling

  • At a cost cap, completed runs retain their held-out outcomes, while runs still in progress count as lacking a final submission and score 0.With task parameters fixed at their FCI fit, the resulting FCI curve cannot fall as the budget grows.Source: §5.3 · p. 11
  • The leading agent changes with the cost budget: DeepSeek V4.1 Flash leads at the smallest budgets, while Claude Opus 5.5 leads at the largest.DeepSeek V4.1 Flash’s costs are lower bounds, so its small-budget lead depends on how its tokens are priced.Source: §5.3 · p. 11

6 Case studies

The case studies show that agents can produce substantial task-specific improvements, but passing and final scores can miss important trade-offs and remaining failures.

6.1 Two checksums shrink private lookups

  • Muse Wonder 1.0 passes QuarterPIR above the human reference, and no other passing run shrinks lookup bytes nearly as much.The measures do not distinguish its larger byte saving; a pass counts the same whether a run saves a few bytes or the most.Source: §6.1 · p. 12, Figure 9 · p. 13

6.2 Two faster clocks for a packet scheduler

  • Only Gemini 4 Argon, Claude Opus 5.5 and GPT-6.1 Sol pass BBQ; both new designs exceed reference throughput on every tree while trading speed for size.Claude moves counters into registers and updates a whole tree level in one cycle; its largest circuit uses the most lookup tables among passing circuits, and it does not serve trees as large as BBQ was designed for.Source: §6.2 · p. 13, Figure 10 · p. 14

6.3 A measured budget for sparse video attention

  • No SVG-EAR run passes; GPT-6 Astra is faster on every clip and closer to full attention on average, yet fails image-quality bands on two clips.Astra measures attention error using real queries every few denoising steps and selects the smallest budget below an error bound for each head.Source: §6.3 · p. 14, Figure 11 · p. 15

7 Conclusion

FrontierCS 2 evaluates agents rebuilding deleted paper contributions in their original repositories, with patch feedback and private final evaluation. Its case studies show that pass and final-score measures miss some consequential side costs, motivating their reporting alongside benchmark outcomes.

7 Conclusion

  • FrontierCS 2 evaluates agents against released paper code after deleting the contribution, with a judge keeping the evaluator, held-out suite and human reference private.Agents can submit patches during the run and receive development-suite scores.Source: §7 · p. 15
  • The case studies show that passes and final scores can miss image-quality failures, large byte savings and circuit-size costs.The authors recommend reporting such side costs next to the pass.Source: §7 · p. 15
  • FCI and FDI place agents and tasks on one scale using metric outcomes and score margins, with intervals indicating which differences the data support.The authors recommend these measures for benchmarks like FrontierCS 2.Source: §7 · p. 15

A Task sources

Task sources are catalogued in Table 2; the set is built from papers’ released code except for the original Structured LWE challenge.

A Task sources

  • Structured LWE is the sole task not built from a paper’s code: agents attack 200 published instances, and any secret meeting the published conditions counts.Because the instances are public, its final suite has no held-out part.Source: A Task sources · p. 24
  • Table 2 lists the paper behind each of the 48 tasks.The three FlashLib tasks share one source, as do the two FARGO tasks.Source: Table 2 · p. 24, A Task sources · p. 24
  • The listed task sources span domains including HPC, programming languages, architecture, cryptography, networking, security, robotics and bioinformatics.For example, QuarterPIR is listed under Efficient Pre-processing PIR Without Public-Key Cryptography, and BBQ under BBQ: A Fast and Scalable Integer Priority Queue for Hardware Packet Scheduling.Source: A Task sources · p. 25

B Task resources

The resource appendix describes the environments and evaluation budgets available for each task, then records agent harness settings used in evaluation.

B Task resources

  • Agents have five hours on every task, and GPU tasks provide short-lived GPU runners for experiments and separate verifier GPUs for scoring.The verifier GPUs are A100 80 GB models.Source: B Task resources · p. 26
  • Table 3 records task resources, including agent cores and memory, GPU type and hours, training evaluations, and final-evaluation time limits.GPU-hour entries are budgets for the whole run, not measured costs; GPU tasks use separate verifier GPUs for scoring.Source: Table 3 · p. 26, B Task resources · p. 26

C Agent settings

  • Table 4 lists harness versions, reasoning effort and output-token limits; some entries are unconfirmed, and settings can vary across calls or shrink to fit context.Muse Code uses low- or medium-effort side calls; Pi and several other harnesses can reduce output limits to the remaining context space.Source: C Agent settings · p. 27

D Computing FCI and FDI

The appendix fits FCI and FDI jointly from agent runs, combining metric outcomes with score margins relative to the human reference. It describes how the fit produces agent and task scales, uncertainty intervals, and cost-budget estimates.

D Computing FCI and FDI

  • 624 runs feed the fit, which groups metrics that runs always beat or miss together into 115 task-specific groups.A run beats a metric only when it beats the reference on every held-out workload; it passes only when it beats every metric.Source: D Computing FCI and FDI · pp. 27–28
  • Score margins convert each run’s distance from the human reference into a common unit, with above-reference failures contributing only a nonnegative margin.The unit is the larger of the task score spread and a floor set to the median spread across tasks.Source: D Computing FCI and FDI · pp. 28–29
  • The run likelihood models agent strength, metric-group difficulty, task tie level, and a run effect, integrating out the run effect.Priors pull groups that no run beats toward the task’s other evidence, keeping their fitted difficulty finite.Source: D Computing FCI and FDI · p. 29
  • The fit jointly estimates 13 agent strengths and 163 task parameters, then fits five hyperparameters using an approximate marginal likelihood.The joint mode uses Newton’s method; the hyperparameter search uses a Laplace approximation and Nelder–Mead.Source: D Computing FCI and FDI · pp. 29–30
  • FCI measures agent strength relative to Kimi K2.7, while FDI locates task difficulty at the strength with a 50% pass probability.Ten FCI points equal one logit; task pass probability requires beating every group in the same run.Source: D Computing FCI and FDI · p. 30
  • At each cost budget, FCI is refit using runs completed within budget; unfinished runs count as having no final submission.The budget fit holds task parameters and hyperparameters fixed and does not use development scores.Source: D Computing FCI and FDI · pp. 30–31

Related papers

Loading 2610.29932v1…