Source-linked AI summary
daVinci-Env: Open SWE Environment Synthesis at Scale
Dayuan Fu, Shenyu Wu, Yunze Wu, Zerui Peng, Yaxing Huang, Jie Sun, Ji Zeng, Mohan Jiang, Lin Zhang, Yukun Li, Jiarui Hu, Liming Liu, Jinlong Hou, Pengfei Liu
TL;DR
SWE-agent training lacks large, diverse, transparent executable environments, and environment quality and difficulty also require systematic curation. OpenSWE provides an open multi-agent synthesis and filtering framework at scale, with models reaching 62.4% and 66.0% on SWE-bench Verified and improving mathematical and science benchmarks without degrading factual recall.
Problem
Open-source SWE training environments remain limited in scale and repository diversity, while industrial systems are opaque and costly to reproduce.
Method
OpenSWE uses a transparent multi-agent pipeline on a 64-node cluster to synthesize executable Docker environments and apply difficulty-aware quality filtering.
Results
OpenSWE-trained models reach 62.4% and 66.0% on SWE-bench Verified, with up to 12-point mathematical-reasoning and 5-point science gains without factual-recall degradation.
Takeaways & Limitations
OpenSWE consistently outperforms SWE-rebench across model sizes and scaffolds, while data scaling and difficulty-aware filtering provide complementary benefits.
Takeaways & Limitations
Construction can be destabilized by nondeterministic LLM latency, network-dependent dependency resolution, synthesized scripts, and resource contention.
Abstract
from arXiv · showhide
Training capable software engineering (SWE) agents demands large-scale, executable, and verifiable environments that provide dynamic feedback loops for iterative code editing, test execution, and solution refinement. However, existing open-source datasets remain limited in scale and repository diversity, while industrial solutions are opaque with unreleased infrastructure, creating a prohibitive barrier for most academic research groups. We present OpenSWE, the largest fully transparent framework for SWE agent training in Python, comprising 45,320 executable Docker environments spanning over 12.8k repositories, with all Dockerfiles, evaluation scripts, and infrastructure fully open-sourced for reproducibility. OpenSWE is built through a multi-agent synthesis pipeline deployed across a 64-node distributed cluster, automating repository exploration, Dockerfile construction, evaluation script generation, and iterative test analysis. Beyond scale, we propose a quality-centric filtering pipeline that characterizes the inherent difficulty of each environment, filtering out instances that are either unsolvable or insufficiently challenging and retaining only those that maximize learning efficiency. With $891K spent on environment construction and an additional $576K on trajectory sampling and difficulty-aware curation, the entire project represents a total investment of approximately $1.47 million, yielding about 13,000 curated trajectories from roughly 9,000 quality guaranteed environments. Extensive experiments validate OpenSWE's effectiveness: OpenSWE-32B and OpenSWE-72B achieve 62.4% and 66.0% on SWE-bench Verified, establishing SOTA among Qwen2.5 series. Moreover, SWE-focused training yields substantial out-of-domain improvements, including up to 12 points on mathematical reasoning and 5 points on science benchmarks, without degrading factual recall.
1. Introduction
OpenSWE addresses limited scale, repository diversity, transparency, and environment-quality challenges in SWE-agent training. It combines large-scale open infrastructure with difficulty-aware curation and reports strong benchmark and out-of-domain results.
- Quality-centric curation: OpenSWE filters environments for PR-Issue alignment and appropriate difficulty, removing unsolvable or insufficiently challenging instances.The curation process targets learning efficiency rather than scale alone.
- Scale and transparency: 45,320 executable environments span 12.8k repositories, with Dockerfiles, evaluation scripts, and the synthesis pipeline fully open-sourced.The framework was built through a multi-agent system deployed across a 64-node cluster.
- Quality-centric curation: About 13,000 curated trajectories are obtained from roughly 9,000 high-quality environments after an additional $576K investment in sampling and curation.Environment construction costs approximately $891K.
- Empirical validation: 62.4% and 66.0% on SWE-bench Verified are achieved by OpenSWE-32B and OpenSWE-72B, respectively, establishing SOTA among Qwen2.5-series methods.The results highlight complementary contributions from data scaling and difficulty-aware curation.
- Empirical validation: Out-of-domain gains reach 12 points on mathematical reasoning and 5 points on science benchmarks without degrading factual recall.The reported results also show log-linear improvement with additional high-quality environments and no saturation.
2. Related Work
Related work establishes executable SWE environments and agent scaffolds as core infrastructure, while Figure 2 illustrates quality risks that OpenSWE seeks to address.
- Task-quality risks: Figure 2 contrasts an unsolvable PR caused by partial commit-hash validation with a trivial PR whose issue reveals the exact edit.These examples illustrate PR-Issue misalignment and insufficient task difficulty.
- Executable SWE environments: SWE-bench pioneered Docker-based repository snapshots with executable test suites serving as evaluation oracles for real GitHub issues and pull requests.Subsequent efforts automate environment generation at larger scale.
- SWE-agent scaffolds: SWE-agent established a foundational scaffold for autonomous repository navigation, bug localization, and patch generation.Agent scaffolds provide infrastructure for multi-step repository interaction.
- Training and synthesis: SWE-smith synthesizes large-scale task instances and execution trajectories for supervised fine-tuning, while daVinci-Dev combines planning with iterative generation and debugging.These approaches address training-data synthesis and agent reasoning through different strategies.
3 Method
OpenSWE constructs Python SWE environments from GitHub pull requests through staged filtering, repository exploration, and multi-agent environment generation.
- Data collection: GitHub PR collection retrieves repository metadata, linked issue descriptions, commit sequences, and corresponding diffs through REST and GraphQL APIs.These records provide the raw material for environment construction.
- Data filtering: Each candidate contains a repository identifier, PR number, associated issues, and the complete PR patch.The filtering process operates on this structured PR dataset.
- Data filtering: The four-stage filter retains repositories with at least five stars, Python as the primary language, nonempty linked issues, and substantive non-test code changes.These criteria target project maturity, language consistency, task specification, and real implementation work.
- Environment synthesis: A multi-agent system converts selected PRs into reproducible Docker containers and validated evaluation scripts capable of confirming solution correctness.The process connects repository exploration with downstream Dockerfile and evaluation-script generation.
- Repository exploration: A repository exploration agent uses metadata and patch-derived file cues to collect bounded setup- and test-relevant evidence from the local checkout.Its constrained interfaces support browsing, searching, and digesting selected configuration and documentation files.
3.4 Dockerfile Construction
OpenSWE constructs Docker environments and evaluation scripts iteratively, using cached layers, repository provisioning, targeted tests, and rule-based validation to improve scalable reproducibility.
- Dockerfile construction: Network instability and redundant rebuilds are identified as recurring Dockerfile-construction failure modes that become costly at scale.The design therefore emphasizes reusable base layers and reduced external dependence.
- Dockerfile construction: Pre-built openswe-python images cover Python 2.7 and 3.5–3.14 with conda, activated testbeds, and package mirrors.They reduce dependency-installation timeouts and enable layer reuse across tasks sharing Python versions.
- Dockerfile construction: Repositories are injected from a local bare-repository cache with target commits checked out in advance instead of cloned during image construction.This removes GitHub rate limits and network failures from the agent loop.
- Dockerfile construction: Stable Dockerfile layers are placed early while dependency installation is isolated into later layers for cheaper iterative rebuilding.The strategy exploits the higher revision frequency of dependency specifications relative to Dockerfile structure.
- Iterative generation: The Dockerfile and evaluation-script agents receive repository context and test-analysis feedback, refining their outputs across iterations.This shared feedback loop supports recovery from environment or test-command errors.
- Evaluation and validation: Evaluation scripts target issue-relevant tests, may synthesize missing cases, and use structured bash output and exit-code markers for reliable validation.The scripts separate patch injection from test-command logic and enforce conda activation where needed.
- Evaluation and validation: A sample is accepted only when test-only changes fail on the unpatched codebase and the full fix patch passes all tests.Docker images are built once per iteration before script execution.
- Infrastructure optimization: Retaining Docker images until the Dockerfile changes yields a 5× speedup when only the evaluation script is revised.Concurrent evaluations are isolated with dedicated CPU, memory, and storage limits.
3.7 Test Analysis
The pipeline validates synthesized environments by analyzing both successful and failed test runs, while a fault-tolerant 64-node framework supports large-scale construction. It produced 45,320 validated environments in approximately two weeks.
- Test Analysis: Passing runs are checked for genuine test execution, including detection of hardcoded exit codes or other evaluation-script shortcuts.
- Test Analysis: Failed runs are diagnosed for Dockerfile misconfiguration, evaluation-script errors, or inherently unsolvable environments.
- Distributed Construction: 64 compute nodes parallelize Docker-based evaluation across isolated nodes while processing approximately 572,114 GitHub PRs.
- Distributed Construction: The framework addresses execution instability from external dependencies and synthesized scripts, plus resource contention causing memory exhaustion or disk saturation.
- Distributed Construction: Independent node operation and a shared-filesystem message queue decouple task production from consumers, limiting the impact of individual node failures.
- Distributed Construction: 45,320 validated environments were completed in approximately two weeks, with the standardized 64-node configuration identified as a near-optimal operating point.
4 Experiments
Experiments show that OpenSWE-trained models achieve strong SWE-Bench Verified performance across model sizes and agent scaffolds. Performance improves with model capacity and training-data scale, while OpenSWE environments consistently outperform SWE-Rebench and transfer to broader capabilities.
- Main Results: 62.4% and 66.0% are achieved by OpenSWE-32B and OpenSWE-72B, respectively, on SWE-Bench Verified.OpenSWE-32B surpasses Qwen2.5-series methods, while OpenSWE-72B outperforms daVinci-Dev-72B by 7.5%.
- Main Results: 3.6% is the gain from OpenSWE-32B to OpenSWE-72B, compared with 2.4% for daVinci-Dev at the same scaffold.The comparison suggests that higher-quality training environments enable better use of increased model capacity.
- Main Results: 59.8% and 62.4% are reached by OpenSWE-32B with OpenHands and SWE-Agent, while OpenSWE-72B reaches 65.0% and 66.0%.SWE-Agent performs consistently better, but both scaffolds benefit from OpenSWE data.
- Data Scaling Analysis: Pass@1 improves approximately log-linearly with training steps across all four model–scaffold configurations.Pearson correlations range from r=0.882 to r=0.972, with no observed saturation within the tested budget.
- Environment Source: 12.2% is OpenSWE's largest reported absolute advantage over SWE-Rebench, at the 32B SWE-Agent setting.OpenSWE leads SWE-Rebench across all four configurations, including by 2.6% in the 72B CodeAct configuration.
- Environment Source: 68.0% is achieved by mixing SWE-Rebench with OpenSWE for 72B SWE-Agent, but 32B SWE-Agent performance falls from 62.4% to 61.4%.The mixed source provides additional gains primarily for larger models and may introduce distribution-shift sensitivity for smaller models.
- Generalization: SWE-focused training improves broader capabilities, with the largest gains appearing on code benchmarks and consistent improvements across three mathematics benchmarks.The evaluation covers code generation, mathematical reasoning, scientific knowledge, and general language understanding.
5 Conclusion
OpenSWE combines transparent, large-scale environment synthesis with difficulty-aware curation to produce stronger training data for SWE agents. Experiments show leading SWE-bench Verified performance and broad gains from curated SWE training.
- 45,320 executable Docker environments across 12.8k repositories are released with all infrastructure open-sourced.The framework includes a multi-agent synthesis pipeline deployed on a 64-node cluster and a quality-centric filtering process.
- Approximately 10,000 high-quality environments are curated by addressing PR-Issue misalignment and triviality.The filtering process is intended to provide a stronger training signal than existing alternatives.
- 62.4% and 66.0% are achieved by OpenSWE-32B and OpenSWE-72B on SWE-bench Verified, establishing state-of-the-art among SFT-based methods.
- OpenSWE-trained models outperform SWE-rebench-trained models across model sizes and scaffolds, with log-linear scaling and broader gains without factual-recall degradation.
A SWE Environment Builder: Architecture and Prompt Excerpts
The builder converts a repository snapshot and patch information into a reproducible Docker environment and evaluation script through retrieval, synthesis, validation, and refinement. Its prompts enforce focused repository exploration, precise environment details, and separated test execution.
- A SWE Environment Builder: Architecture and Prompt Excerpts: The builder takes a repository snapshot at a fixed base commit with patch information and produces a Dockerfile plus a machine-readable bash evaluation script.
- A SWE Environment Builder: Architecture and Prompt Excerpts: The iterative loop retrieves context, synthesizes artifacts, builds and executes the environment, extracts structured log markers, and refines failures.
- A.1 Prompt Design: Prompt design enforces engineering invariants for stable, large-scale synthesis rather than merely documenting implementation details.
- A.1 Prompt Design: Repository exploration is goal-driven and non-exhaustive, producing a concise report with exact dependency versions and concrete test commands.
- A.1 Prompt Design: Retrieval covers versioned dependencies, language runtimes, configuration files, OS requirements, precise tests, and CI/CD instructions.
- A.1 Prompt Design: Agents retrieve only explicitly requested files when possible, using limited exploration otherwise and avoiding redundant or speculative searches.
- A.1 Prompt Design: Findings are organized for downstream agents into actionable setup and testing guidance, including tools, commands, variables, and relevant configurations.
- A.1 Prompt Design: Dockerfile generation requires repository checkout, environment setup, dependency installation, test configuration, shell compatibility, and no tests during image construction.
B. Construction Cost Estimate SII-GAIR
The construction-cost workflow includes structured agent guidance for handling environment and evaluation failures. A documented limitation is that some dependencies are unavailable or excessively old.
- Some dependencies are missing or unavailable, including 404 files and very old versions such as numpy <= 1.8.
- The output format records completion status and separate guidance for Dockerfile, evaluation-script, and context-retrieval agents.
- Context retrieval should be requested only when missing information is necessary to fix a Dockerfile or evaluation-script issue.
B Construction Cost Estimate
The cost estimate covers a 64-node, 10-day construction run and additional trajectory sampling and difficulty-aware curation. Reported costs are approximate and sensitive to pricing and utilization assumptions.
- Construction cost depends primarily on effective GPU-hour pricing and cluster utilization efficiency.Preemptible pricing, committed-use discounts, and scheduling efficiency can substantially change the final amount.
- Approximately $576,000 is additionally required for trajectory sampling and difficulty-aware curation.The cost includes LLM API expenses and Docker compute for trajectories across OpenSWE and SWE-rebench datasets.
- A 64-node, 10-day run underlies the approximate construction-cost estimate.