PaperlayerView exact revision on arXiv

Source-linked AI summary

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Hantian Ding, Chloe Bi, Jiacheng Zhu, John Yang, Matt Deitke, Pengcheng Yin, Zijian Wang, Rui Hou

arXiv:2609.38335v1cs.SEcs.AI

TL;DR

Whole-repository generation calls for system-level planning, but evaluating it requires specifications and tests that support implementation-independent solutions. E2E-SWE constructs and iteratively audits a multilingual benchmark for that task; across 13 frontier models, pass@1 ranges from 11.7% to 67.7%.

  • Problem

    Whole-repository generation requires system-level planning and integration, while benchmark results are meaningful only when every tested behavior is specified and implementation-independent solutions can pass.

    Source: §1 · p. 2

  • Method

    E2E-SWE pairs engineer- and LLM-created behavioral tests with an implementation-independent specification, then audits tasks statically and against real agent rollouts.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    Pass@1 ranges from 11.7% to 67.7% across 13 frontier models on E2E-SWE.

    Source: §1 · p. 2

  • Takeaways & Limitations

    E2E-SWE provides a demanding, measurable test of whole-repository generation, with clear model separation and substantial headroom.

    Source: §6 · p. 10

    The post-hoc judge may overlook behaviors implied by requirements distributed across the specification and misclassify genuine model errors as task defects.

    Source: §4.1 · pp. 7–8

Abstract

from arXiv · show

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

1 Introduction

Whole-repository generation demands coordinated, long-horizon planning, yet evaluating it requires tasks whose hidden tests faithfully reflect the specification. E2E-SWE builds and audits such tasks, and its results distinguish current models while leaving substantial headroom.

1 Introduction

  • Whole-repository development requires system-level planning, decomposing requirements into components, coordinating interfaces, and integrating and debugging the system against many requirements.The challenge extends beyond implementing components individually to making them work together.Source: §1 · p. 2
  • E2E-SWE asks agents to build complete, installable repositories from a natural-language specification in an empty workspace, while hidden tests check user-facing behavior.Specifications and tests are created from scratch with engineer and LLM collaboration, leaving internal architecture to the agent; the benchmark isolates autonomous implementation from the preceding specification-writing stage.Source: §1 · p. 2
  • Because full resolution requires every hidden test to pass, an underspecified behavior or overly rigid test can prevent a task from being solvable.The authors define solvability as every tested behavior being derivable from the specification, with any implementation satisfying the user-facing contract able to pass.Source: §1 · p. 2, Figure 2 · p. 3
  • Static specification-test audits and rollout-guided reviews distinguish genuine implementation errors from task defects missed during inspection.The review loop examines failures from real model attempts and repairs identified defects.Source: §1 · p. 2
  • Pass@1 ranges from 11.7% to 67.7% across 13 frontier models on the final E2E-SWE dataset.The dataset contains 186 tasks across 11 programming languages, drawn from reference repositories with a median of 7.4K lines of code.Source: §1 · p. 2

2 The E2E-SWE Benchmark

E2E-SWE tasks pair implementation-independent behavioral tests with specifications that document their requirements, then undergo static and rollout-guided review. Agents implement these projects from scratch, and the corpus spans varied languages and software functions.

2.1 Task Formulation

  • Agents receive only a specification in an empty workspace and must create a repository plus setup.sh that installs it; grading runs the installer and hidden tests in a fresh container.Network access is disabled during implementation and grading, and the test entrypoint reports every test outcome in language-agnostic CTRF format.Source: §2.1 · p. 3

2.2 Task Creation

  • Each task starts with an engineer and LLM creating the test suite and specification, with the author reviewing, steering, and making design decisions.The LLM drafts specifications and test suites that may run to hundreds or thousands of lines.Source: §2.2 · p. 4

2.2.1 Building the Test Suite

  • Tests are newly authored around realistic user-facing scenarios and observable behavior, avoiding dependence on private helpers or other reference-code structure.Authors use original tests to identify intended behaviors and edge cases, and measure coverage without maximizing it indiscriminately.Source: §2.2.1 · p. 4

2.2.2 Writing the Specification

  • Specifications are written after the tests and document their required behaviors while leaving untested implementation choices open.Authors audit specification-test alignment assertion by assertion, making open choices explicit or relaxing tests to accept reasonable alternatives.Source: §2.2.2 · p. 4

2.3 Quality Refinement

  • Static reviewers assess test quality and specification fairness, while rollout-guided review examines agent failures to distinguish implementation errors from task defects.The automated refinement loop uses specialized reviewers and a revision agent; observing generated implementations can expose defects that static inspection misses.Source: §2.3 · p. 4

2.4 Dataset Statistics

  • The corpus contains 186 tasks across 11 programming languages and 13 functional categories, spanning diverse software domains.Figure 3 presents corpus composition by language and functional category; task complexity also reflects both specification breadth and reference-repository size and maturity.Source: §2.4 · p. 5, Figure 3 · p. 5

3 Experiments

Frontier models vary substantially in full-repository resolution, while greater reasoning effort improves pass@1 up to saturation for the tested models.

3.1 Experimental Setup

  • Thirteen frontier models are evaluated across six model families, using each model’s strongest reasoning-effort setting that reliably fits the 128K per-request output-token limit.Some models at max effort may exceed the limit and cause API call failures; each model is run independently four times over the full corpus.Source: §3.1 · p. 5
  • All models use a simple ReAct agent with a four-hour rollout budget; the primary metric, pass@1, requires passing every hidden test in one rollout.There is no partial credit: full resolution weights tasks equally rather than rewarding a fraction of tests passed.Source: §3.1 · pp. 5–6

3.2 Main Results

  • Claude Opus 5 achieves the highest pass@1, followed by Claude Fable 5 and GPT-5.6 Sol, while even the strongest model leaves nearly one third of tasks unresolved.Performance is broadly distributed, giving E2E-SWE strong distinguishing power among frontier models.Source: Table 2 · p. 6, §3.2 · p. 6
  • Fable 5 nearly matches Opus 5 using roughly two thirds as many output tokens, while Opus 4.6 resolves fewer tasks than GPT-5.6 Sol despite using over three times as many turns.Sonnet 5 uses almost as many tokens as Opus 5 but resolves substantially fewer tasks.Source: §3.2 · p. 6
  • E2E-SWE elicits more output tokens than DeepSWE and FrontierCode for every matched model, but turn counts are similar to DeepSWE.The paper attributes the similar turn counts largely to E2E-SWE providing the full specification upfront, removing initial codebase exploration.Source: §3.2 · p. 6, Figure 4 · p. 7
  • From low through xhigh effort, Opus 4.8 and GPT-5.5 use more output tokens and achieve consistently higher pass@1; Opus 4.8’s gains saturate at max.Moving Opus 4.8 from xhigh to max consumes considerably more tokens without further improving pass@1; only four tasks hit the 128K limit, too few to change this pattern.Source: §3.2 · p. 6, Figure 5 · p. 7

4 Analysis

The analysis examines task-defect attributions, generated repository size, reasoning patterns, and reference-code overlap. It finds that codebase size is weakly related to success, reasoning often peaks early, and substantial code overlap does not ensure task resolution.

4.1 Task Solvability

  • 29 of 744 attempts (3.9%) had a failure attributed to the task; excluding those failures would change 16 outcomes (2.2%) from unresolved to resolved.The audits examine rollout failures to distinguish model implementation errors from task defects; these rates are judge attributions, not definitive defect counts.Source: §4.1 · pp. 7–8

4.2 Generated Codebases

  • Generated repositories are generally smaller than references, and success shows no strong relationship to generated LOC; size appears to reflect coding style more than correctness.Claude models more often generate codebases at the larger end of the range, consistent with their higher output-token usage; Opus 4.6 and Sonnet 5 generate similarly sized codebases despite a large pass@1 gap.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Reasoning Distribution

  • For 11 of 13 models, the longest reasoning turn typically falls within the first 12 turns; for six models, it falls within the first four.Peak reasoning can exceed 100K tokens at P90, and peak share exceeds 50% in the most concentrated cases. The authors attribute the early peak to receiving the full specification upfront without an existing codebase to explore.Source: §4.3 · p. 8

4.4 Memorization

  • Claude Fable 5, Opus 5, and Opus 4.8 exceed 50% containment on 24–30 tasks and 75% on 8–13, yet high overlap does not guarantee resolution.For tasks above 50% containment, the resolve-rate deltas are −0.6 pp, +9.6 pp, and +5.3 pp for Fable 5, Opus 5, and Opus 4.8, respectively.Source: §4.4 · pp. 8–9

5 Related Work

Prior repository benchmarks have assessed localized edits and varied whole-repository tasks, but the related work identifies a gap in domain-general generation from a project specification alone. E2E-SWE is distinguished by stating tested behaviors upfront and pairing specifications with independently authored tests.

5 Related Work

  • RepoBench, RepoCoder, CrossCodeEval, and DevEval evaluate line- or function-level completion, while SWE-bench and variants target PR-level changes in existing repositories.These settings assume the project is largely implemented.Source: §5 · p. 9
  • Whole-repository benchmarks cover varied formulations and domains, but none provides domain-general evaluation of repository creation from a project specification alone.Examples include staged development, reconstruction from signatures and docstrings, API scenarios, and domain-specific software tasks.Source: §5 · p. 9
  • NL2Repo-Bench is the closest prior benchmark, but its scores cluster near 10% for most models and provide limited frontier-model separation.E2E-SWE instead pairs independently authored tests with test-driven specifications and iteratively reviews their behavioral alignment.Source: §5 · pp. 9–10
  • ProgramBench and MirrorCode combine behavior discovery by probing with implementation, whereas E2E-SWE states tested behaviors upfront.The former also require relevant repository functionality to be exposed through a standalone, queryable program.Source: §5 · p. 10

6 Conclusion

E2E-SWE evaluates complete repository generation from natural-language specifications with independently created tests and iterative review. Across 13 models, it yields a broad range of pass@1 scores, supporting strict but meaningful evaluation of this long-horizon task.

6 Conclusion

  • E2E-SWE comprises 186 repository-generation tasks across 11 programming languages, with specifications and hidden tests created from scratch and iteratively reviewed and repaired.The design preserves freedom in internal implementation while making full-repository resolution meaningful.Source: §6 · p. 10
  • Across 13 models, pass@1 ranges from 11.7% to 67.7%, yielding clear model separation and substantial headroom.A post-hoc LLM audit attributes outcome-changing failures to task issues in 2.2% of attempts.Source: §6 · p. 10
  • The authors conclude that E2E-SWE offers a demanding yet measurable test of long-horizon software-engineering capability in whole-repository generation.Source: §6 · p. 10

A Evaluation on NL2Repo-Bench

The appendix evaluates the models on NL2Repo-Bench using the same scaffold and reasoning-effort settings, with network access disabled. Scores are tightly clustered, and the authors suggest that factors beyond task complexity limit the benchmark’s ability to distinguish frontier models.

A Evaluation on NL2Repo-Bench

  • All 13 models are evaluated on NL2Repo-Bench with the same agent scaffold and reasoning-effort settings, while network access is disabled to prevent retrieval of public reference implementations.Rollouts start with required dependencies pre-installed.Source: A Evaluation on NL2Repo-Bench · p. 14
  • Except for Claude Opus 5 and Fable 5, models resolve only 8–15 tasks, with many scores separated by one or two tasks.Pass@1 is lower than on E2E-SWE for every model.Source: A Evaluation on NL2Repo-Bench · p. 14, Table 4 · p. 14
  • Smaller median reference repositories and fewer output tokens suggest NL2Repo-Bench’s compressed scores reflect factors beyond task complexity, limiting frontier-model resolution.The reported median repository size is 1.5K–4K LOC, below E2E-SWE’s 7,360 LOC median.Source: A Evaluation on NL2Repo-Bench · p. 14

Related papers

Loading 2609.38335v1…