PaperlayerView exact revision on arXiv

Source-linked AI summary

NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents

Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, Ge Zhang

arXiv:2512.12730v2cs.CL

TL;DR

Existing coding-agent benchmarks provide limited evidence about whether agents can sustain the planning and coherence needed to construct complete repositories. NL2Repo-Bench evaluates repository generation from a requirements document and empty workspace; across agents, repository generation remains difficult, with Claude-Sonnet-4.5 achieving a 39.6% pass rate.

  • Problem

    Existing benchmarks emphasize localized generation, scaffolded completion, or repair, leaving autonomous repository construction over hundreds of steps inadequately measured.

    Source: §1 · pp. 1–2

  • Method

    NL2Repo-Bench evaluates agents tasked with building a functional Python library from a single requirements document and an empty workspace.

    Source: §1 · p. 2

  • Results

    All agents struggled with long-horizon planning and cross-file consistency, while Claude-Sonnet-4.5 achieved a 39.6% pass rate.

    Source: §5 · p. 18

  • Takeaways & Limitations

    The authors argue that progress will require innovations in agentic planning, self-correction, and environment management.

    Source: §5 · p. 18

    Task solvability depends on the requirements document's API Usage Guide completely and accurately describing every function exercised by the tests.

    Source: §3.1.2 · p. 6

Abstract

from arXiv · show

Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.

1 Introduction

NL2Repo-Bench evaluates whether coding agents can build complete Python libraries from a single requirements document and empty workspace. Experiments show that repository generation remains difficult, with even the strongest agents achieving below 40% average test pass rates.

  • Prior benchmarks often test functions, module completion, or repair, while scaffolding, existing code, limited scope, and subjective evaluation leave full-repository generation from scratch underexplored.Source: §1 · pp. 1–2
  • NL2Repo-Bench provides only a requirements document and empty workspace, then tests the resulting installable Python library against an upstream pytest suite in a controlled environment.It comprises 104 tasks across diverse domains and complexity levels, with requirements averaging nearly 19k tokens.Source: §1 · pp. 2–3
  • Even the strongest tested agents achieve below 40% average test pass rates and rarely reproduce repositories, with failures spanning premature termination, lost architectural coherence, brittle dependencies, and weak persistent execution and verification.Source: §1 · p. 3
  • NL2Repo-Bench adds a long-horizon evaluation axis complementary to function-level and repair benchmarks, and points to planning, self-verification, and global-coherence mechanisms as research needs.Source: §1 · p. 3
  • The benchmark formalizes repository construction from an empty workspace and contributes a quality-assured task corpus and standardized evaluation image for apples-to-apples agent comparisons.Source: §1 · p. 3

2 Related Works

Prior repository-generation benchmarks vary in their input priors, evaluation methods, and outputs, while coding agents and code-oriented models are mostly evaluated on existing-codebase or function-level tasks. NL2Repo-Bench instead tests complete repository construction from a single natural-language document using official upstream tests.

2.1 Coding Benchmarks for LLMs

  • NL2Repo-Bench evaluates repository generation from a single natural-language document without scaffolding or signatures, using the upstream repository’s official pytest suite as its sole evaluator.Prior work spans repair/regression, completion, paper-to-repository reproduction, and from-scratch generation with project structures or function signatures; these differ in input priors, evaluation judges, and required outputs.Source: §2.1 · p. 3

2.2 LLM-Driven Coding Agents

  • SWE-agent and OpenHands support repository-level coding and autonomous development in existing codebases, whereas NL2Repo-Bench tests complete repository construction from a single document.SWE-agent’s Agent–Computer Interface abstracts file search, navigation, editing, and testing; OpenHands supports planning, coding, and validation with minimal human intervention.Source: §2.2 · pp. 3–4

2.3 Coding-Oriented LLMs

  • Code Llama [22], OpenCoder [11], Seed-Coder [36], and Seed-Thinking [23] advance code understanding and generation, but most evaluations still emphasize function-level tasks.The passage also identifies StarCoder-style and DeepSeek-style model families, long-context support up to 100k tokens, domain knowledge distillation, and self-curated data synthesis as developments in coding-oriented models.Source: §2.3 · p. 4

3 NL2Repo-Bench

NL2Repo-Bench evaluates whether agents can reconstruct complete Python repositories from a single natural-language specification, with correctness measured against upstream tests. Its construction pipeline selects, documents, configures, and validates realistic repository-generation tasks.

3 NL2Repo-Bench

  • Agents receive only a natural-language specification and must reconstruct a complete repository, with correctness measured by running the original upstream tests.Tasks derive from real-world Python libraries with modular architectures and authoritative pytest suites.Source: §3 · p. 4

3.1.1 Repository Selection

  • Target repositories must meet size, maturity, testability, and recency criteria, and annotators retain only projects that pass their native tests.Projects contain 300–120,000 lines of code, have at least 10 GitHub stars, include passing pytest suites, and were created or updated within three years.Source: §3.1.1 · p. 4

3.1.2 Project Document Writing

  • Annotators reverse-engineer each repository into a structured specification, using AST inventories to document APIs and their expected behavior exhaustively.Each document includes a project description, development supports, an API usage guide, and implementation nodes; omissions or misaligned API definitions can make tasks unresolvable.Source: §3.1.2 · pp. 5–6

3.1.3 Environment Building

  • Each task receives a Docker-based environment whose dependencies are iteratively adjusted until the upstream repository passes its official tests.Build manifests are also sanitized to remove non-functional prerequisites, so evaluation focuses on generated software functionality.Source: §3.1.3 · p. 6

3.1.4 Verification and Refinement

  • Tasks undergo expert review, AST-based API coverage checks, and agent feasibility trials, followed by refinement until every verification stage passes.A senior engineer diagnoses test failures and exceptions against the documentation and environment, addressing ambiguities or configuration artifacts so failures reflect model capabilities.Source: §3.1.4 · pp. 6–7

3.2 Benchmark Statistics

  • 104 tasks across nine Python-library categories average approximately 18,800 input tokens, establishing a large-scale benchmark for repository generation from natural-language descriptions.The benchmark also categorizes tasks as easy, medium, or hard based on original project size and lines of code; detailed difficulty statistics are presented in Table 2.Source: Table 1 · p. 7, §3.2 · p. 7, Table 2 · p. 7

4 Experiments

The experiments assess coding agents’ repository-generation performance across tasks, examining how model capability, execution, planning, context retention, task complexity, and interaction rounds relate to outcomes. They also probe whether performance bottlenecks persist when agents are given the complete test suite rather than hidden requirements.

4.1 Experimental Settings

  • Agents start from an empty workspace and one specification, work autonomously without a round limit, and are scored by the original repository’s pytest pass rate.All collected tests are run even if collection errors occur, preventing a few such errors from automatically assigning a zero score.Source: §4.1 · p. 8

4.2 Main Results

  • Below 40.5% average test pass rate, every model falls short; the strongest fully passes only five repositories in one run.Nearly half the models score below 20%.Source: Figure 3 · p. 8, Table 3 · p. 9, §4.2 · p. 9
  • Claude leads the model comparisons, performance declines with task difficulty, and results vary by category while changing less than 1% across Claude-Sonnet-4.5’s three frameworks.Claude’s advantage is associated with its larger context window and ability to sustain long interaction traces. Claude-Sonnet-4.5 is particularly strong on system tools and database tasks, while all models struggle more on machine learning and networking; inter-model gaps exceed framework variation.Source: §4.2 · pp. 9–10

4.3.1 Tools Used During Development

  • Task-tracker use correlates most strongly with model score among the three most-used tools, while editing dominates tool calls and planning usage differs across models.The task-tracker correlation is 0.711; code editing accounts for 48–62% of calls, and edit-to-execution ratios are 1.67:1 for Claude-Sonnet-4.5, 1.62:1 for GPT-5, and 1.51:1 for DeepSeek-V3.2. GPT-5 allocates 14.5% of calls to planning, Claude-Sonnet-4 14.2%, and Qwen3-Thinking 0; call quantity is not a monotonic predictor of score.Source: §4.3.1 · p. 10, §4.3.4 · pp. 13–14

4.3.2 Interaction Turns and Model Performance

  • GPT-5 averages 78.4 interaction turns versus Claude-4.5’s 181.6 and often stops for user input; early termination and non-finish distinguish incomplete-task behaviors.GPT-5 has a 13.4% early-termination rate and an 84.5% non-finish rate, whereas Qwen3-Thinking terminates early in 49.0% of tasks and Claude-Sonnet-4 has a 1.9% non-finish rate and 0 early stops. More turns alone do not ensure success: Claude-4.5 has the best turns/score ratio (455.1), while DeepSeek-V3.2 uses 254.3 turns for a 27.6% score.Source: §4.3.2 · pp. 11–12, §4.3.3 · pp. 12–13

4.3.5 Impact of Context Window Size

  • A 1M+ context window supports repository performance but is not sufficient without strong reasoning and persistent agent behavior.Claude-Sonnet-4.5 scores 40.2% on Claude Code and Gemini-3-pro 34.2%; Kimi-k2 scores 22.7%, below 128K DeepSeek-V3.2 at 27.6%. Long tasks combine 18,800-token average specifications, 10,000–50,000-token codebases, and about 90,000 tokens of conversation history; context also helps preserve planning and execution state.Source: §4.3.5 · pp. 14–16

4.3.6 Agentic Workflow Patterns

  • High-performing agents cycle between editing and testing, while weaker workflows include navigation without edits and consecutive edits without tests.Repository failures also stem from broken package structure or internal dependencies and from mismatches between implementations and official test expectations.Source: §4.3.6 · p. 16, §4.3.7 · p. 16

4.4.1 Impact of interaction round limits

  • Performance rises as interaction limits increase from 50 to 200 rounds, then improves only marginally, pointing to reasoning and coherence limits beyond the round budget.The 200-round threshold is slightly above Claude-Sonnet-4.5’s unrestricted average; remaining limitations include semantic reasoning, architectural planning, and cross-file consistency.Source: §4.4.1 · pp. 16–17, Figure 9 · p. 17

4.4.2 Impact of revealing all test cases.

  • By default, agents receive only the natural-language specification; the experiment compares this hidden-test setting with generation where all test cases are visible.The comparison is designed to distinguish difficulty inferring implicit requirements from deeper implementation challenges that persist under full supervision.Source: §4.4.2 · p. 17
  • With all tests exposed, Claude-Sonnet-4.5 (Claude Code) pass rate rose from 40.2% to 59.4%, and fully-passed tasks from 3 to 18; its overall score remained below 60%.The authors interpret the remaining gap as evidence that repository generation remains challenging even with test-driven development, reflecting limitations in long-horizon coordination and large-scale code synthesis.Source: §4.4.2 · p. 17

5 Conclusion · Appendix

NL2Repo-Bench evaluates whether LLMs and agents can build complete, installable Python repositories from a single natural-language document. Experiments show that repository-level generation remains challenging, motivating advances in agent planning and reliability.

5 Conclusion

  • The benchmark evaluates repository generation from scratch, starting with a single natural-language document and requiring a complete, installable Python repository.Source: §5 · p. 17
  • Separating development from evaluation and using authoritative upstream test suites provides a strict measure of autonomous software engineering capability.Source: §5 · p. 18
  • Claude-Sonnet-4.5 achieved a 39.6% pass rate, while agents broadly struggled with long-horizon planning and cross-file consistency.The conclusion describes a substantial gap between current state-of-the-art models and repository-level generation requirements.Source: §5 · p. 18
  • Two critical failure modes are thinking models’ overconfidence causing early termination and GPT-5’s collaborative bias preventing autonomous progress.Source: §5 · p. 18
  • Progress will require architectural innovations in agentic planning, self-correction loops, and reliable environment management, not just larger context windows.Source: §5 · p. 18
  • NL2Repo-Bench is intended as a testbed for the next generation of coding agents.Source: §5 · p. 18
  • The released benchmark includes the dataset, Docker environments, and evaluation toolkit for the research community.Source: §5 · p. 18

A Difficulty Level in NL2Repo Tasks · B Available Tools in OpenHands CodeAct Framework

NL2Repo task difficulty is classified by the original repository’s lines of code, while experiments use standardized tools from the OpenHands CodeAct framework. The tools support command execution, planning, editing, information lookup, task tracking, and submission simulation.

A Difficulty Level in NL2Repo Tasks

  • Difficulty is based on original repository LOC: fewer than 1,500 is Easy, more than 4,000 is Hard, and intermediate sizes are Medium.The paper states that Table 2 presents these thresholds.Source: A Difficulty Level in NL2Repo Tasks · p. 23

B Available Tools in OpenHands CodeAct Framework

C Tutorial for NL2Repo-Bench Annotators

The section outlines the annotator workflow for constructing NL2Repo task specifications that faithfully represent repositories, support end-to-end development, and remain consistent across annotators.

C Tutorial for NL2Repo-Bench Annotators

  • Annotation proceeds through project selection, repository comprehension and environment validation, then structured specification writing.These practical guidelines aim to ensure specifications are semantically faithful, comprehensive for end-to-end repository development, and consistent across annotators.Source: C Tutorial for NL2Repo-Bench Annotators · p. 23

C.1 Phase 1: Project Selection

Phase 1 screens candidate GitHub repositories for eligibility before inclusion.

C.1 Phase 1: Project Selection

  • Annotators assess project maturity, testing completeness, license compatibility, and whether the repository’s core functionality can be isolated.Eligibility follows the criteria in Section 3.1.1 of the main paper.Source: §C.1 · p. 23

C.2 Phase 2: Repository Understanding & Test Validation

Phase 2 gives annotators a precise, executable understanding of the target repository before specification, then checks that its environment and tests are usable.

(1) Local Setup and Preliminary Analysis

  • Annotators download the repository from GitHub and review its purpose, core functionality, directory structure, and external dependencies before writing its specification.This review is intended to establish a precise and executable understanding of the target repository.Source: §C.2 · p. 24

(2) Environment Construction

  • Annotators create an isolated environment using repository documentation and install all relevant dependencies, including potentially undocumented ones.Documentation may include the README, requirements.txt, setup.py, or pyproject.toml.Source: §C.2 · p. 24

(3) Full Test Execution

  • A repository is valid only when all existing tests pass; annotators diagnose environment-related failures, and repositories with unresolvable failures are excluded.Examples of environment issues include Python-version mismatches, incompatible dependency versions, and missing system packages.Source: §C.2 · p. 24

C.3 Phase 3: Task Specification Construction

Phase 3 constructs a comprehensive specification for each task.

C.3 Phase 3: Task Specification Construction

  • Each specification combines a project-level description, support information, and an API-level usage guide.Source: §C.3 · p. 24

C.3.1 Project Description … C.4 Implemention Nodes

The annotation process gives coding agents a manually verified account of a repository’s purpose, structure, dependencies, functional units, API usage, and critical-API examples. It combines high-level summaries with implementation-aligned documentation and repository-derived import guidance.

C.3.1 Project Description

  • Annotators summarize the project’s purpose, design, major components, and interactions, then manually verify the final description for factual correctness.An initial summary may draw on official documentation or LLM-based tools.Source: §C.3.1 · p. 24

C.3.2 Support Information

  • Support information lists required third-party libraries, including test dependencies, and preserves version numbers where applicable.Source: §C.3.2 · p. 24
  • The support information includes the complete implementation-relevant repository layout while excluding the testing directory.Source: §C.3.2 · p. 24

C.3.3 API Usage Guide

  • Static analysis inventories classes, functions, and constants with their names, signatures, and file locations, excluding test directories.Source: §C.3.3 · p. 24
  • For each functional node, annotators document its name, purpose, arguments, returns, and implementation-aligned behavior, then manually verify details against source code.AI tools may assist with drafting.Source: §C.3.3 · pp. 24–25
  • Module and API import examples follow internal import patterns observed in test files and omit imports from external libraries.Source: §C.3.3 · p. 25

C.4 Implemention Nodes

  • Annotators provide repository-specific examples or references for critical APIs when available to clarify their usage.They may consult examples in the repository.Source: §C.4 · p. 25

C.5 Quality Requirements

C.5 requires descriptions to faithfully and completely specify repository behavior, without unsupported or omitted functionality, so an agent can implement the repository from the instructions alone.

C.5 Quality Requirements

  • Descriptions must match the repository implementation and cover every functional node.These requirements make the specification both faithful and comprehensive.Source: §C.5 · p. 25
  • Descriptions must not include speculative or missing behaviors that are not grounded in the source code.Source: §C.5 · p. 25
  • The resulting specification should enable an agent to implement the entire repository solely from the provided instructions.Source: §C.5 · p. 25

Related papers

Loading 2512.12730v2…