PaperlayerView exact revision on arXiv

Source-linked AI summary

DeNovoSWE: Scaling Long-Horizon Environments for Generating Entire Repositories from Scratch

Jiale Zhao, Guoxin Chen, Fanzhe Meng, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia

arXiv:2606.10728v2cs.SE

TL;DR

Long-horizon repository generation lacks large-scale, verifiable training data. DeNovoSWE automatically constructs 4,818 document-to-repository instances through sandboxed divide-and-conquer, critic-repair, and difficulty-aware filtering, raising Qwen3-30B-A3B from 5.8% to 47.2% on BeyondSWE-Doc2Repo.

  • Problem

    Long-horizon repository generation lacks large-scale, verifiable training data for agents implementing complete repositories from documentation.

  • Method

    DeNovoSWE uses an automated sandboxed pipeline combining divide-and-conquer documentation generation, iterative critic-repair, and difficulty-aware trajectory filtering.

  • Results

    47.2% on BeyondSWE-Doc2Repo, versus 5.8% originally, after fine-tuning Qwen3-30B-A3B on DeNovoSWE.

  • Takeaways & Limitations

    DeNovoSWE provides effective long-horizon training data for whole-repository generation across Qwen3 and Qwen3.5 backbones.

  • Takeaways & Limitations

    The difficulty estimator assumes harder instances have lower rollout pass rates, requiring a negative correlation between estimated difficulty and pass rate.

Abstract

from arXiv · show

As the capabilities of LLM-based code agents continue to advance, their expected role is expanding beyond localized bug fixing in existing codebases toward architecting and implementing complete software repositories from high-level specifications. However, training agents for such long-horizon software engineering tasks remains difficult due to the scarcity of large-scale, verifiable whole-repository generation data. In this paper, we introduce \textbf{DeNovoSWE}, a large-scale dataset for whole-repository generation. DeNovoSWE comprises 4,818 high-quality instances, where each instance requires generating a complete repository from documentation. Our dataset is automatically constructed through a carefully designed sandboxed agentic workflow, enabling scalable curation without human annotation. DeNovoSWE is constructed with "divide and conquer" and critic-repair philosophy. To balance data quality and diversity, we further introduce a difficulty-aware trajectory filtering strategy. Fine-tuning Qwen3-30B-A3B on DeNovoSWE substantially improves long-horizon SWE performance, raising its score on the challenging BeyondSWE-Doc2Repo benchmark from 5.8% to 47.2%.

1. Introduction

DeNovoSWE addresses the lack of scalable, verifiable data for long-horizon whole-repository generation, a setting requiring planning and interdependent coding beyond single-issue fixes. It introduces an automated sandboxed pipeline and difficulty-aware filtering to construct diverse, high-quality document-to-repository tasks at scale.

  • Existing SWE training datasets largely target single-issue fixes, whereas whole-repository generation requires long-horizon planning and complex, interdependent coding.
  • Whole-repository generation requires solutions for documentation construction, evaluation design, and leakage-free task execution.Repository documentation may be incomplete, unstructured, or misaligned with executable behavior, making it insufficient for directly defining tasks.
  • DeNovoSWE introduces an automated sandboxed pipeline for generating complete repositories from comprehensive, organized, behavior-consistent documentation.The framework uses a divide-and-conquer paradigm to synthesize documentation aligned with repository behavior and executable evaluation.
  • 4,818 high-quality instances comprise DeNovoSWE-Data, a large-scale dataset for verifiable whole-repository generation.
  • Difficulty-aware trajectory filtering balances data quality and task diversity while curating verifiable expert trajectories for complex repository-generation tasks.

2. Related Work

Prior work has expanded software engineering benchmarks toward multimodal, multilingual, and long-horizon evaluation while introducing repository-level datasets and specialized models or agent frameworks.

  • SWE Benchmark: SWE benchmarks have broadened from SWE-bench and SWE-bench-Verified to assess multimodal, multilingual, and long-horizon software engineering capabilities.The cited passage identifies these benchmark directions as emerging after the prevailing SWE-bench benchmarks.
  • SWE Datasets: Repository-level SWE datasets address increasingly complex coding tasks, including real-world data construction in SWE-Gym and large-scale datasets such as Scale-SWE, OpenSWE, and SWE-rebench.The passage frames high-quality data as pivotal for improving LLM programming capabilities.
  • SWE Models: Recent SWE models include SWE-RL, SWE-Swiss, SWE-World, SWE-Master, and Orchard, alongside scaffolds such as SWE-agent, Mini-SWE-Agent, OpenHands, OpenComputer, and MOpenHands.These models and frameworks are presented as advances for specialized SWE tasks and agentic workflows.

3. DeNovoSWE: Scaling Long-Horizon Repository Generation

DeNovoSWE constructs verifiable document-to-repository tasks through sandboxed multi-agent decomposition, profiling, mapping, and iterative critic-repair documentation. Its 4,818 instances provide broad scale, high executable coverage, and substantial repository-level complexity.

  • Workflow design: The sandboxed workflow generates comprehensive, organized, executable documentation containing sufficient behavioral and structural information for implementations to pass repository evaluation tests.This requirement is more demanding than ordinary documentation synthesis.
  • Divide phase: Repository decomposition assigns functional capabilities to relevant modules, functions, classes, and interfaces, reducing the complexity of repository-level documentation generation.The process combines capability extraction, repository profiling, and mapping through an LLM-based classifier.
  • Conquer phase: Three sandboxed agents iteratively draft, critique, and repair capability-level documents before merging them with the repository overview into final task documentation.Critiques check component coverage, APIs, import paths, input-output behavior, and usage constraints.
  • Dataset scale: 4,818 instances make DeNovoSWE over an order of magnitude larger than NL2RepoBench’s 104 and BeyondSWE-Doc2Repo’s 50 repository-generation tasks.The larger scale is intended to increase task diversity relative to existing benchmarks.
  • Dataset statistics: 85.5% average coverage, 89.6% median coverage, 96.5% P75 coverage, and 99.9% P90 coverage characterize DeNovoSWE’s executable test coverage.These statistics indicate that selected repositories are generally well constrained by unit tests.
  • Dataset statistics: 9 median source files, 21 at P75, 42 at P90, 79 median unit tests, and 12 test files show that evaluation spans interacting repository components.The statistics indicate repository-level complexity beyond isolated functions.

4. Difficulty-Aware Trajectory Filtering

DeNovoSWE uses difficulty-aware trajectory filtering because strict global pass criteria discard valuable partial solutions in long-horizon repository generation. It estimates instance difficulty from structural and LLM-based signals, then assigns stricter thresholds to easy tasks and more permissive thresholds to hard ones.

  • Motivation: Whole-repository generation produces trajectories with varying quality because errors compound across files, APIs, dependencies, and implementation details.Requiring trajectories to pass all unit tests is therefore overly restrictive for document-to-repository generation.
  • Difficulty Estimation: The difficulty estimator combines executable Python line counts, two independent five-level LLM difficulty judgments, and empirical rollout pass rates.Harder instances are expected to have lower rollout pass rates, so estimated difficulty should correlate negatively with the pass-rate signal.
  • Difficulty Estimation: The method normalizes heterogeneous signals to [0, 1] and fuses them into a unified difficulty score using a convex combination.The structural signal uses log(1 + ·) followed by percentile-based min-max normalization, while LLM ordinal levels are mapped linearly.
  • Dynamic Filtering: Difficulty-aware filtering partitions the continuous difficulty range into five uniform intervals of width 0.2 and assigns each interval an instance-specific trajectory threshold.This dynamic scheme preserves informative trajectories from challenging repositories while reducing the selection bias of a fixed global threshold.
  • Dynamic Filtering: 0.90 is the threshold for difficulty scores in [0.0, 0.2), whereas 0.60 is used for scores in [0.8, 1.0].The threshold decreases monotonically with difficulty: easy tasks receive stricter quality control, while hard tasks retain partial but informative solutions.

5. Experiments

Experiments show that DeNovoSWE substantially improves open-weight agents on whole-repository generation, approaching proprietary frontier performance. Difficulty-aware trajectory filtering further improves downstream results over fixed-threshold alternatives.

  • Main results: 5.8% and 4.3% are the original Qwen3-30B-A3B-Instruct scores on BeyondSWE-Doc2Repo and NL2RepoBench, rising to 29.2% and 18.3% with Scale-SWE-Agent.Issue-level SWE data transfers usefully but only limitedly to whole-repository generation.
  • Main results: 43.8% and 23.5% are the original Qwen3.5-35B-A3B scores, increasing to 50.0% and 27.1% after DeNovoSWE fine-tuning.The gains across Qwen3 and Qwen3.5 backbones support DeNovoSWE as effective long-horizon training data.
  • Filtering ablation: Lowering the fixed trajectory-score threshold from 0.95 to 0.60 consistently improves both BeyondSWE-Doc2Repo and NL2RepoBench performance.This indicates that overly strict global filtering can remove useful trajectories from challenging repository-generation tasks.
  • Filtering ablation: 0.488 to 0.500 and 0.264 to 0.271 are the gains from difficulty-aware filtering on BeyondSWE-Doc2Repo and NL2RepoBench, respectively.The strategy preserves valuable hard-instance trajectories while applying stricter quality control to easier repositories.

6. Conclusion

DeNovoSWE is introduced as a large-scale real-world dataset for document-to-repository software engineering, targeting long-horizon code-agent training. Its structured divide-and-conquer and iterative critic-repair pipeline automatically produces 4,818 high-quality repository-generation instances.

  • 6. Conclusion: DeNovoSWE provides a large-scale real-world dataset for document-to-repository software engineering tasks and long-horizon code-agent training.It addresses the lack of verifiable training data for whole-repository generation.
  • 6. Conclusion: 4,818 high-quality repository-generation instances are automatically constructed through a structured divide-and-conquer pipeline.The pipeline is designed for scalable dataset construction.
  • 6. Conclusion: An iterative critic-repair mechanism supports the automatic construction of high-quality repository-generation instances.This mechanism complements the divide-and-conquer pipeline.

A. DeNovoSWE data structure

This section specifies the structure of the DeNovoSWE dataset.

  • Table 5 presents the DeNovoSWE data structure specification.

B. Hyperparameter

The section specifies that SFT hyperparameters are provided in Table 6 and that standardized evaluation configurations across BeyondSWE-Doc2Repo and NL2Repo-Bench are provided in Table 7.

  • SFT hyperparameters: SFT hyperparameters are detailed in Table 6.The passage identifies Table 6 as the source for the SFT settings.
  • Evaluation hyperparameters: Evaluation uses a uniform set of core hyperparameters for all models across BeyondSWE-Doc2Repo and NL2Repo-Bench, documented in Table 7.The configuration is intended to ensure reproducible and standardized evaluations across both benchmarks.

C. License Filtering.

DeNovoSWE filters repositories by license, excluding unknown, missing, restrictive, or otherwise unsuitable licenses. It retains repositories under permissive licenses appropriate for research-oriented model training.

  • License Filtering: DeNovoSWE excludes repositories with unknown, missing, restrictive, or otherwise unsuitable licenses during repository selection.The filtering is license-aware and targets repositories unsuitable for training-data construction.
  • License Filtering: It retains repositories licensed under permissive schemes including MIT, Apache-2.0, BSD-family licenses, ISC, and CC0-1.0.Other accepted licenses include 0BSD, Unlicense, Zlib, PostgreSQL, NCSA, Boost-1.0, BSL-1.0, and Python-2.0.

D. Prompt for DeNovoSWE data construction.

The DeNovoSWE construction prompt guides agents to produce evidence-grounded repository overviews from documentation and repository artifacts. It specifies exploration, documentation, interface-verification, and structured-output constraints to make the resulting descriptions coherent and reliable.

  • Evidence and exploration: Agents must ground claims in the README, repository tree, capability outline, package metadata, entry points, source packages, and repository evidence.They should inspect the codebase whenever the repository’s role or architecture remains unclear.
  • Output constraints: Outputs must be submitted through the structured `finish` tool rather than as free-form chat prose, with final checks for overall purpose, evidence support, documentation style, and completion.The ability prompt likewise requires structured completion and forbids writing documentation into repository files.
  • Evidence and exploration: The prompt prohibits speculative or unsupported interfaces, empty adjectives, narrow function-level phrasing, capability-bullet restatement, and writing the overview into repository files.Mentioned interfaces such as CLIs, APIs, endpoints, configuration systems, and plugin surfaces require supporting evidence.
  • Overview writing rules: The prompt requires a natural `## 1. Overview` section explaining the repository’s purpose, domain, interface type, and major workflows or subsystems.It targets roughly 120-350 words and prioritizes documentation-worthy behavior over helper internals.
  • Ability-specific documentation: For ability-specific documentation, agents must cover exactly one coherent feature or workflow, verify signatures and behavior in code and tests, and preserve qualified symbol and highlighted API names verbatim.Exact contracts such as errors, payload keys, defaults, constants, mode strings, and configuration keys should be documented when behavior depends on them.
Loading 2606.10728v2…