PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Alex Gu, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang

arXiv:2609.38269v3cs.SEcs.AI

TL;DR

Benchmarks for from-scratch repository construction are limited, while common evaluations start from short programs or mature repositories. Zero2Repo builds tasks from pinned projects and evaluates complete repositories against hidden tests; its strongest configuration solves 15 of 17 tasks, while the release remains small and uneven across languages.

  • Problem

    Many coding benchmarks evaluate short programs or patches to mature repositories, leaving repository architecture, dependencies, and build systems already in place.

    Source: §1 · pp. 1–2

  • Method

    Zero2Repo turns pinned open-source projects into public specifications, reproducible environments, and hidden tests, then scores complete repositories with deterministic all-tests-pass evaluation.

    Source: §1 · pp. 2–3

  • Results

    15 of 17 tasks are solved by the strongest configuration, and no configuration solves every task.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The benchmark separates current systems, and its failures include near misses as well as runs that never receive a complete judge result.

    Source: §6 · p. 12

    The 17-task release is small and imbalanced across languages, and results on its 11 well-known seed projects may overstate performance on unseen repositories.

    Source: §6 · p. 12

Abstract

from arXiv · show

Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements document, an interface contract, and an empty workspace, and must deliver a complete repository in the project's native ecosystem. Tasks are produced by a language-agnostic authoring pipeline that converts real, version-pinned open-source projects into behavioral specifications, reproducible environments, and hidden acceptance tests. Each task is validated by execution: a reference implementation derived from the upstream project must pass, and adversarial validation must show that the tests reject incorrect implementations. Evaluation runs production coding agents in isolated containers, withholds the acceptance tests until an explicit submission, and assigns a binary reward only when every test passes, with no LLM judge. The pipeline and harness make no language-specific assumptions and apply to mainstream programming ecosystems; the current release contains Python, TypeScript, Go, and C++ tasks. Even on 11 tasks drawn from repositories that frontier models have very likely seen during training, the strongest agent solves only 10, and every failing submission passes 90-99% of the hidden tests; for the two strongest agents, 67-100% of failed tests trace to a single omission or a low-frequency rule stated in the specification rather than to a missing subsystem, so each failure is a concrete target for improvement.

1 Introduction

Existing benchmarks largely test short programs or patches to repositories whose architecture and build systems already exist. Zero2Repo instead asks agents to build complete native-ecosystem repositories from specifications, and its strongest configuration still leaves tasks unsolved.

1 Introduction

  • Repository delivery requires translating product-level descriptions into architecture, interfaces, dependencies, and build configuration, unlike short-program or patch evaluations.Those evaluations begin with the architecture, dependency graph, and build system already in place.Source: §1 · pp. 1–2
  • Zero2Repo gives agents a PRD, Interface Contract, reproducible toolchain, and empty workspace, while leaving architecture and implementation to them.Credit requires every hidden acceptance test to pass, without an LLM judge.Source: §1 · p. 2
  • 15 of 17 tasks are solved by the strongest configuration, yet no configuration solves every task.Both failures of the leading configuration pass at least 96% of hidden tests.Source: §1 · p. 3
  • Tasks derive from pinned open-source projects and undergo reference execution and adversarial validation before release.The pipeline checks executable behavior, rebuildable environments, and tests that reject incorrect implementations; human review inspects the resulting artifacts.Source: §1 · pp. 2–3
  • A common submission and judging protocol distinguishes judged near misses, timeouts, and missing submissions; one task defeats every configuration.The evaluation covers 18 configurations.Source: §1 · p. 3
  • The same task assets support supervised trajectories and reinforcement-learning environments with the hidden suite as a verifiable reward.The outcome split distinguishes judged submissions with missing coverage from runs that never reach the judge.Source: §1 · p. 3

2 Related Work

Prior work evaluates short programs, repository patches, and several kinds of from-scratch development, but differs in starting artifacts and success criteria. Zero2Repo targets complete repositories with native packaging and build artifacts, sealed acceptance tests, and automatically authored tasks validated before release.

Local code generation and repository modification

  • Short-program and repository-completion benchmarks leave architecture, dependencies, and build systems in place, while the SWE-bench family tests issue resolution in existing repositories.RepoBench and CrossCodeEval add repository context but retain completion as the target.Source: §2 · p. 3

From-scratch software construction

  • From-scratch benchmarks differ in starting artifacts and success criteria; Zero2Repo requires native packaging and build artifacts with sealed acceptance tests.Other approaches include stage-wise checks, Python library scaffolds, restored reference packaging, microservices, and output-equivalence evaluation.Source: §2 · p. 3

Long-horizon agent evaluation and task construction

  • Zero2Repo automatically authors specifications, interfaces, environments, and tests, releasing tasks only after reference execution and adversarial validation.This contrasts with expert-curated long-horizon tasks and task construction that relies on replayed user feedback.Source: §2 · p. 4

3 The Zero2Repo Benchmark

Zero2Repo gives agents a public behavioral specification and an empty workspace, then evaluates their completed repository against hidden tests. Its tasks come from pinned projects and pass execution, adversarial, and review checks before release.

3.1 Task Formulation

  • Agents receive a public specification and installed toolchain in an empty workspace, choose their architecture, and must submit a repository that passes every hidden test.The PRD specifies behavior; the Interface Contract fixes package names, APIs, CLI entry points, argument conventions, return types, and error types.Source: §3.1 · pp. 4–5

3.2 Dataset Construction and Verification

  • Pinned repositories supply the behavior for generated specifications and tests, while release requires a rebuildable environment, a passing reference implementation, adversarial rejection checks, and review.The 17-task release spans multiple ecosystems but is not balanced across languages.Source: §3.2 · p. 6

3.3 Evaluation Framework

  • After explicit submission, the workspace is copied to a fresh judge container with hidden tests; reward is one only when tests run and all pass.The judge does not run installation or build commands, so required artifacts must already be in the workspace.Source: §3.3 · p. 7

4 Experiments

The 17-task evaluation compares repository delivery across 18 model–harness configurations using an all-tests-pass reward. No configuration solves every task; results vary by task and harness, and higher time or cost does not consistently correspond to more solved tasks.

4.1 Setup

  • The evaluation reports pass rate as the fraction of 17 tasks with reward 1, alongside agent time, token totals, and published API spend.Each configuration represents one adopted high-reasoning run; solve and judge phases each have a 7200-second limit.Source: §4.1 · p. 8

4.2 Main Results

  • GPT-6 Astra with Codex solves 15 of 17 tasks, ahead of Claude Opus 5.5 with OpenCode at 14; no configuration solves every task.A task counts as solved only when every hidden test passes.Source: §4.2 · p. 8, Table 2 · p. 9
  • Of 306 runs, 134 pass and 172 do not; 121 unsuccessful runs are judged failures, while 51 never receive a complete judge result.The 51 include 28 timeouts, 19 exits without submission, three token-limit stops, and one interrupted test run.Source: §4.2 · p. 9
  • The leading configuration uses 4 hours 13 minutes and $80.57, while the second-place configuration uses 9 hours 35 minutes and $195.69.Qwen3.8 Max records the longest total, 29 hours 21 minutes, and solves 6 tasks.Source: §4.2 · p. 9
  • Holding the model fixed, harness-associated pass-count differences range from one task for Claude Sonnet 5.5 to three for GPT-6 Astra, Claude Opus 5.5, and GPT-6 Sol.Because these are separate adopted runs, the differences mix harness behavior with run-to-run variation.Source: §4.2 · p. 9
  • Task difficulty varies: task 004 is solved by 17 configurations, task 014 by none, and task 016 only by GPT-6 Astra with Codex.The leading configuration solves every task labeled Python, both Go tasks, and the C++, Zig, and one of two TypeScript and C tasks; the small language samples do not support language rankings.Source: §4.2 · p. 11

5 Failure Analysis

Most judged failures are near misses, but many unsuccessful runs never receive a complete judge result. Failures also differ across tasks, while the benchmark’s outcome split distinguishes missed behavior from runs that do not reach judging.

5 Failure Analysis

  • Among 121 submitted runs judged short of a full pass, 95 passed at least 90% of hidden tests and 80 passed at least 95%; 26 passed less than 90%.The two strongest configurations’ five unsuccessful tasks passed at least 94% of hidden tests. On task 014, judged scores ranged from 151/320 to 316/320.Source: §5 · p. 11
  • Fifty-one of 172 unsuccessful runs receive no complete judge result: 28 time out, 19 exit without submission, three hit the token limit, and one has interrupted tests.On task 014, five configurations time out or never submit, so their scores do not measure how much of the navigation filter they implemented.Source: §5 · p. 11
  • Failure patterns differ by task: task 016 is solved by one configuration, task 006 by two, and tasks 003, 009, and 010 by three each.The configurations solving tasks 003 and 009 are not the same, so a single unsolved-task list would conceal that variation.Source: §5 · p. 11
  • The outcome split distinguishes judged submissions missing specified behavior from runs that time out or fail to submit, although the release lacks per-test labels for adopted submissions.The hidden suite can support diagnostic rewards, but the full-repository all-pass reward treats these different outcomes alike.Source: §5 · p. 12

6 Conclusion, Limitations, and Future Work

Zero2Repo evaluates complete native-ecosystem repository construction and distinguishes systems, while its authoring pipeline supports expansion across languages. The conclusion also notes limits in task scale and evaluation coverage, alongside plans for future versions and further evaluation.

6 Conclusion, Limitations, and Future Work

  • Across 17 tasks and 18 configurations, GPT-6 Astra with Codex solves 15, Claude Opus 5.5 with OpenCode solves 14, and the lowest pass rate is 1 of 17.The same model in a second harness changes its score by as many as three tasks; judged failures are usually near misses, while 51 runs lack a complete judge result.Source: §6 · p. 12
  • The release has 17 tasks, with nine in Python and only one or two tasks in each other language; one adopted run per configuration-task pair leaves run-to-run variance unmeasured.The first 11 seed projects are well known, so their pass rates are probably higher than rates on unseen repositories.Source: §6 · p. 12

AI use statement

  • Generative AI drafted task artifacts and manuscript materials, but released task artifacts passed execution checks and adversarial validation and were inspected by the authors.Authors checked the paper's claims, numbers, and analyses against experiment records.Source: AI use statement · pp. 12–13

Ethics statement

  • The benchmark uses permissively licensed open-source projects, neutralizes identifying information, and versions future releases to limit exposure in training data.The pipeline is not released; task artifacts, harness, and experiment scripts are available, and environments rebuild from public base images and recipe locks.Source: Ethics statement · p. 13, Reproducibility statement · p. 13

B Task Package Layout

Task packages separate public specifications from private runner metadata, environment recipes, and acceptance tests. Agents must build the complete project in the empty workspace, validate it themselves, and explicitly submit the final state for hidden testing.

B Task Package Layout

  • The package provides the PRD and Interface Contract to the agent, while runner metadata, environment recipes, denylist, and final hidden tests are not present in the agent image.Optional hardware constraints and declared external resources are also listed in the package layout.Source: B Task Package Layout · p. 15

C Agent Instruction

  • Agents must implement the specified project from scratch, read both specification files, and leave the workspace ready for the judge to use directly.The agent instruction expects iterative development, testing, and debugging; general-purpose libraries are allowed, but downloading or installing the target implementation is prohibited.Source: C Agent Instruction · p. 16
  • A valid submit file is required to trigger hidden acceptance tests, which run against the final workspace and return a binary pass/fail reward.The judge does not install or build the project; the provided build contract specifies whether a build step is required and where outputs must remain.Source: C Agent Instruction · pp. 15–17

D Example Specification Excerpt

The excerpt illustrates how a PRD defines task terminology and behavior, while the Interface Contract specifies callable inputs and errors. Its float-converter example distinguishes normal conversion behavior from invalid converter results.

D Example Specification Excerpt

  • The PRD defines a float converter as a caller-supplied callable for constructing TOML floats and specifies that it does not change how integers, strings, booleans, date-times, tables, or arrays are handled.FP-05 applies to both string-parse and binary-file parse entries, with Python float as the default when no converter is supplied.Source: D Example Specification Excerpt · p. 17
  • With a decimal converter, 0.982492 becomes a decimal from its characters, while an integer remains a Python integer and a TOML float becomes a decimal.Integers are not passed through the converter.Source: D Example Specification Excerpt · p. 17
  • The Interface Contract requires `s` to be TOML text and refuses bytes, booleans, files, and other non-text values with TypeError rather than TOMLDecodeError.The string-parse entry accepts the document as a positional Python text string.Source: D Example Specification Excerpt · pp. 17–18
  • If the float converter returns a dictionary or list, including a subtype, parsing fails with ValueError rather than TOMLDecodeError.The excerpt distinguishes this illegal converter result from invalid TOML.Source: D Example Specification Excerpt · pp. 17–18

E Validation Before Release

Before release, tasks undergo repeated execution-based validation and author inspection, including checks that reference implementations pass and adversarial variants fail. The process is automated for new tasks but still requires author review.

E Validation Before Release

  • Release checks require the adapted upstream reference to pass the full suite and adversarially altered or hollow implementations to be rejected.Review also checks run integrity, specification-to-test support, independently computed expected values, and that the environment does not leak the upstream implementation.Source: E Validation Before Release · p. 18
  • Among the first 11 tasks, none needed more than four validation rounds before release.Each round used a frontier coding agent under the release harness, and the resulting run was reviewed.Source: E Validation Before Release · p. 18, Table 4 · p. 18
  • Validation runs automatically for every new task, but author inspection still takes about one to two hours per task.The authors expect task production to become fully automatic as agents improve or stronger, more expensive reviewers are used.Source: E Validation Before Release · p. 18

F Agent Integration, Versions, and Budgets

The runner supports four coding-agent systems, with each evaluated system treated as a model–harness configuration. Runs use specified solve and judge budgets, while some run details are not published.

F Agent Integration, Versions, and Budgets

G Additional Results

Table 5 summarizes task-level solve counts across 18 configurations and reports the hidden-test suite size for adopted runs. The paper directs readers to an online source for per-task metrics and explains how it handles differing totals.

G Additional Results

  • Table 5 reports hidden-test suite sizes on adopted runs and the number of configurations with reward 1 for each task.Source: Table 5 · p. 19
  • The paper publishes per-task pass and failure fractions, token totals, and spend online rather than repeating that matrix.Source: G Additional Results · p. 19
  • When published configuration totals differ from sums of per-task figures, §4.1 states which total Table 2 uses.Source: G Additional Results · p. 19
Loading 2609.38269v3…