Source-linked AI summary
NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, Ge Zhang
TL;DR
Existing benchmarks provide limited evidence about whether coding agents can sustain coherent planning and execution while constructing complete repositories from natural language. NL2Repo-Bench evaluates this capability from an empty workspace using test-verified repository generation, and experiments show that it remains largely unsolved, with even the strongest agents achieving a 39.6% pass rate and exhibiting systematic long-horizon failures.
Problem
Existing evaluations emphasize localized generation, scaffolded completion, or short-term repair, leaving full-repository construction from a single natural-language document underexplored.
Method
NL2Repo-Bench gives agents one requirements document and an empty workspace, requiring autonomous architecture design, dependency management, multi-file implementation, and packaging evaluated with upstream tests.
Results
39.6% pass rate is achieved by Claude-Sonnet-4.5, while all agents struggle with long-horizon planning and cross-file consistency.
Takeaways & Limitations
NL2Repo-Bench provides a rigorous testbed showing that reliable repository-level generation requires advances in planning, self-correction, and environment management.
Takeaways & Limitations
The API Usage Guide must completely and accurately describe every functional node exercised by the original tests, because omissions or misaligned definitions make tasks unresolvable.
Abstract
from arXiv · showhide
Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents.
1 Introduction
Existing coding-agent benchmarks underrepresent the sustained planning and cross-file consistency required to build complete repositories from natural language. NL2Repo-Bench addresses this gap with autonomous, test-verified repository generation and shows that current agents remain unreliable over long horizons.
- Existing benchmarks mainly test isolated functions, scaffolded completion, or repair, reducing demands for long-term planning and system-level coherence.These settings often provide structural priors, limited temporal scope, or existing codebases rather than requiring autonomous construction.
- NL2Repo-Bench gives agents one natural-language requirements document and an empty workspace, requiring architecture design, dependency management, multi-file implementation, and packaging.The benchmark covers 104 tasks with diverse domains and complexity levels, and documents averaging nearly 19k tokens.
- Long-horizon repository generation remains largely unsolved: even the strongest agents achieve average test pass rates below 40% and rarely reproduce repositories fully.Observed failure modes include premature termination, loss of global architectural consistency, brittle dependency handling, and weak persistence in executing and verifying plans.
- NL2Repo-Bench highlights agentic planning, self-verification, and global-coherence mechanisms as needed directions beyond larger context windows.It is positioned as both a diagnostic tool for current systems and a benchmark for future autonomous software-engineering research.
- The benchmark evaluates complete installable Python libraries by running generated code against authoritative upstream pytest suites rather than relying on qualitative judgments.This formalizes repository generation as a strictly verifiable long-horizon coding task.
2 Related Works
Prior repository-level benchmarks evaluate repair, completion, or agent operation within existing codebases, while NL2Repo-Bench targets full repository construction from a single natural-language document. Its distinguishing axes are minimal input priors, authoritative testing, and fully installable package output.
- Existing repository benchmarks commonly evaluate issue resolution, component completion, or autonomous work within pre-existing codebases rather than construction from scratch.SWE-agent and OpenHands operate on repository-level tasks but generally assume an existing codebase.
- NL2Repo-Bench differs from prior paradigms by requiring repository-level generation from a single natural-language document without scaffolding or signatures.Prior work includes repair/regression in existing projects and completion of missing components in incomplete repositories.
- Repository benchmarks differ in input priors, evaluation judges, and output forms, with NL2Repo-Bench using a single document, authoritative upstream tests, and fully installable packages.These axes distinguish autonomous generation from edits within a codebase or completion of scaffolded components.
- Code foundation models and autonomous coding frameworks provide the technical basis for repository-level generation, but prior systems do not by themselves evaluate complete construction from natural language.The benchmark focuses this capability on architecture design and packaging.
3 NL2Repo-Bench
NL2Repo-Bench derives reproducible tasks from real Python libraries, documents their behavior through structured natural-language specifications, and verifies generated repositories against upstream tests. Human review, AST-based coverage checks, controlled environments, and iterative refinement support task fidelity.
- Repository Selection: NL2Repo-Bench uses real Python libraries with modular architectures and authoritative pytest suites, measuring generated repositories by executing them against the original tests.This separates development from evaluation and provides verifiable behavioral ground truth.
- Repository Selection: Target repositories are selected for meaningful complexity, maturity, testability, and recency, then retained only after their native tests pass successfully.The selection pipeline excludes trivial or unverifiable projects and checks repository stability before inclusion.
- Project Document Writing: Each task specification reconstructs repository implementation, module relationships, and test logic so an agent can reproduce behavior without source-code access.The resulting document is intended as a high-level functional specification.
- Project Document Writing: Specifications contain project description, supports, API usage, and implementation nodes, with API documentation covering tested functional elements and expected behavior.AST-assisted annotation extracts names, signatures, locations, inputs, outputs, and semantics to guide reconstruction.
- Verification and Refinement: A Docker-based execution environment is configured against official dependencies and refined through ground-truth test verification to support deterministic evaluation.Build manifests are also relaxed where non-functional checks, such as documentation-file existence, would otherwise interfere with functional assessment.
4 Experiments
Across NL2Repo-Bench experiments, agents struggle to generate complete repositories reliably, with performance shaped by model capability, task complexity, persistence, and planning behavior.
- Main Results: All models average below 40.5% test pass rates, with nearly half below 20%, and the strongest fully passes only five of 104 repositories in Pass@1.These results indicate insufficient robustness, long-horizon planning, and cross-file consistency for repository generation from scratch.
- Main Results: Claude models outperform other evaluated models, associated with context windows up to 1M tokens and sustained interaction traces that preserve prior decisions.The comparison links Claude’s advantage to both context capacity and maintaining long interaction histories.
- Main Results: Performance declines substantially as task difficulty increases, exposing weaknesses in long-horizon reasoning, multi-module coordination, and dependency-rich workflows.The monotonic decline supports the benchmark’s difficulty hierarchy for repository-level software development.
- Main Results: Claude-Sonnet-4.5 varies by less than 1% across three agent frameworks, while larger inter-model gaps indicate underlying LLM capability dominates framework effects.The benchmark therefore behaves primarily as a model-centric evaluation of reasoning and generation ability.
- Tool Use and Planning: Task-tracker usage correlates most strongly with performance at 0.711, while Qwen3-Thinking makes no planning-tool calls and exhibits premature termination and incomplete implementations.The paper treats explicit planning as a notable differentiator, while cautioning that correlation does not establish causation.
- Interaction Turns and Model Performance: GPT-5 averages 78.4 interaction turns versus Claude-4.5’s 181.6 and frequently halts for user confirmation, producing a moderate 0.217 score despite high per-turn quality.Its low turn count and premature stopping align better with human-in-the-loop assistance than fully autonomous repository generation.
- Interaction Turns and Model Performance: Claude4.5 balances efficiency and performance at a 455.1 turns/score ratio, whereas DeepSeek-V3.2 uses 254.3 turns for a third-best 27.6% score.The comparison indicates that simply increasing interaction attempts does not guarantee better repository-generation outcomes.
- Failure Modes: Claude-Sonnet-4 achieves 1.9% Non-Finish and 0 Early Stop rates, while Qwen3-Thinking terminates early in 49.0% of tasks and GPT-5 has an 84.5% Non-Finish rate.These contrasting behaviors distinguish persistent completion from premature confidence or dependence on user guidance.
5 Conclusion
NL2Repo-Bench rigorously evaluates complete repository generation from a single natural-language document and exposes substantial limitations in current coding agents.
- NL2Repo-Bench evaluates complete, installable Python repository generation from scratch using authoritative upstream tests.The benchmark starts from a single natural-language document and separates development from evaluation.
- 39.6% pass rate for Claude-Sonnet-4.5 still leaves a substantial gap between current SOTA models and repository-level generation requirements.All agents struggle with long-horizon planning and cross-file consistency.
- The benchmark identifies overconfidence-driven early termination and GPT-5’s collaborative bias as two critical failure modes.These behaviors respectively cause premature stopping and failure to proceed autonomously.
- NL2Repo-Bench is positioned as a testbed for future coding agents and released with its dataset, Docker environments, and evaluation toolkit.The authors argue that progress requires innovations in planning, self-correction, and environment management.
6 Contributions
The supplied contribution section contains contributor and sponsor listings rather than substantive research contributions.
- The section lists the paper’s leading authors, core contributors, contributors, and sponsor committee.No methodological or empirical contribution is stated in these passages.
A Difficulty Level in NL2Repo Tasks
NL2Repo task difficulty is categorized by repository size, while agents use standardized tools for coding, planning, execution, browsing, and task completion.
- Repositories under 1,500 LOC are Easy, those over 4,000 LOC are Hard, and intermediate repositories are Medium.
- The standardized environment includes tools for shell execution, reasoning, completion signaling, browsing, interactive Python, task tracking, editing, fetching, and pull-request simulation.These tools support navigation, package installation, testing, planning, and file modification.
- Task management and precise string-based editing are provided through task_tracker and str_replace_editor.
- The fetch tool retrieves content from a specified URL and can optionally extract it as Markdown.
- The create_pr/create_mr tools simulate pull-request or merge-request submission on common code-hosting platforms.
C Tutorial for NL2Repo-Bench Annotators
The annotation workflow aims to produce faithful, comprehensive, and consistent NL2Repo task specifications through three structured phases.
- Annotators select projects, comprehend repositories and validate environments, then write structured specifications.The workflow is designed to support end-to-end repository development while remaining faithful to the original repository.
C.1 Phase 1: Project Selection
Annotators first determine whether a candidate GitHub repository is eligible for inclusion using project maturity, testing completeness, license compatibility, and core-functionality isolation criteria.
- Repository inclusion depends on project maturity, testing completeness, license compatibility, and feasible isolation of the core functionality.
- The selection phase establishes eligibility before subsequent repository understanding and specification construction.
C.2 Phase 2: Repository Understanding & Test Validation
Annotators build an executable understanding of each eligible repository by reviewing its structure and dependencies, constructing an isolated environment, and validating that all tests pass.
- Annotators review the repository’s purpose, core functionality, directory structure, and external dependencies after downloading it locally.
- They construct an isolated environment from project documentation and install explicit and undocumented dependencies needed for execution.
- Repository validity requires all existing tests to pass, with environment-related failures diagnosed and resolved when possible.
- Repositories with unresolvable test or environment failures are excluded from the benchmark.
C.3 Phase 3: Task Specification Construction
The task specification combines project-level context, support information, and API-level guidance, requiring complete, implementation-faithful coverage of repository structure, dependencies, functional nodes, and usage.
- Each task specification contains a project description, support information, and an API-level usage guide.
- The project description summarizes the repository’s purpose, design, major components, and interaction patterns.
- Support information lists required third-party libraries with applicable versions and provides the complete implementation file structure.
- API guidance documents functional units, including their names, signatures, locations, purposes, arguments, return values, and import paths.
- Critical APIs receive concrete examples or repository references, while descriptions are manually verified against the implementation.
C.5 Quality Requirements
Quality requirements ensure that the specification is implementation-consistent, complete across functional nodes, and free of unsupported behavior, enabling repository reconstruction from the instructions.
- All specification descriptions must remain fully consistent with the repository’s implementation.
- The specification must cover all functional nodes rather than omitting implementation-relevant elements.
- Descriptions must exclude speculative or source-ungrounded behaviors.
- Together, these requirements make the specification comprehensive and faithful enough for an agent to implement the entire repository from the instructions alone.