Source-linked AI summary
Immersion in the GitHub Universe: Scaling Coding Agents to Mastery
Jiale Zhao, Guoxin Chen, Fanzhe Meng, Minghao Li, Jie Chen, Hui Xu, Yongshuai Sun, Wayne Xin Zhao, Ruihua Song, Yuan Zhang, Peng Wang, Cheng Chen, Jirong Wen, Kai Jia
TL;DR
Scale-SWE targets the shortage of large-scale, high-quality SWE training data and the difficulty of curating executable tasks. Its sandboxed multi-agent workflow processes real-world pull requests into verified instances, and fine-tuning on distilled trajectories produces an agent that reaches a 64% resolve rate on SWE-Bench-Verified versus 22% for its base model.
Problem
High-quality SWE training data is scarce because constructing executable environments, comprehensive tests, and self-contained task descriptions is difficult at scale.
Method
Scale-SWE coordinates specialized agents for environment setup, unit-test generation, and problem-description synthesis while processing real-world pull requests in a sandboxed workflow.
Results
64% resolve rate is achieved on SWE-Bench-Verified after fine-tuning Scale-SWE-Agent, up from 22% for Qwen3-30B-A3B-Instruct.
Takeaways & Limitations
Scale-SWE-Data provides a large, diverse, realistic source of verified SWE instances that supports training improved code agents.
Abstract
from arXiv · showhide
Achieving mastery in real world software engineering tasks is fundamentally bottlenecked by the scarcity of large scale, high quality training data. Scaling such data has been limited by the complexity of environment setup, unit test generation, and problem statement curation. In this paper, we propose ScaleSWE, an automated, sandboxed multi agent workflow designed to construct high quality SWE data at scale. The system coordinates three specialized agents for environment setup, test creation, and problem description synthesis to process 6 million pull requests across 5200 repositories, producing Scale SWE Data: 100k verified SWE instances, the largest such dataset to date. It substantially surpasses existing real world datasets in repository diversity and reflects realistic task complexity. We further demonstrate the dataset utility for training by distilling 71498 high quality trajectories and finetuning Qwen30BA3BInstruct to produce ScaleSWE Agent. Our agent achieves a 64 resolve rate on SWE Bench Verified a nearly three fold improvement over the base model. ScaleSWE provides a scalable, reproducible approach for data construction to advance LLM based software engineering. Scale SWE will be publicly available.
1. Introduction
Scale-SWE addresses the shortage of high-quality SWE training data by automating environment setup, test generation, and problem-description synthesis. Processing millions of real-world pull requests yields a large verified dataset and supports improved code-agent training.
- Motivation: High-quality SWE data remains scarce because executable environments, dependencies, and test suites make systematic curation difficult.Existing construction methods are often manual, simplistic, or rule-based, limiting dataset scale, diversity, difficulty, or executability.
- Challenges: Automatic SWE dataset construction must handle heterogeneous environments, insufficient unit tests, and pull requests with incomplete or unsuitable task descriptions.These challenges require interactive execution, self-correction, and repository-context exploration.
- Approach: Scale-SWE coordinates three specialized agents for environment setup, unit-test generation, and self-contained problem-description synthesis.The workflow is sandboxed and designed for scalable, high-quality dataset construction.
- Dataset: 6 million pull requests across 5,200 repositories produce 100,000 verified instances in Scale-SWE-Data, the largest SWE dataset to date.The dataset surpasses prior real-world datasets in repository diversity and reflects realistic software-engineering complexity.
2. Scale-SWE: Software Task Scaling
Scale-SWE uses a sandboxed multi-agent workflow to construct executable SWE tasks by coordinating environment setup, test generation, and problem-description synthesis. Its design supports repository-scale automation while aligning task descriptions with validation tests.
- Workflow: Scale-SWE coordinates environment building, unit-test creation, and problem-statement writing to automate executable SWE task construction.The workflow uses Docker-based environments, generated F2P and P2P tests, and test-grounded task descriptions.
- Environment Builder Agent: The Environment Builder Agent transforms a generic Docker environment into a repository-specific runtime by analyzing dependencies, configuration files, and terminal feedback.The resulting container provides an isolated, reproducible environment for task validation.
- Environment Builder Agent: At most ten pull requests per repository receive full environment construction, allowing shared runtime states to support additional instances while controlling computational costs.Pull requests that fail unit tests in their nearest available environment are discarded; each repository contributes 19 test instances on average.
- Unit-test Creator Agent: The Unit-test Creator Agent combines pull-request metadata, repository context, and the functional Docker environment to generate executable F2P and P2P tests.F2P tests initially fail and must pass after the fix, while P2P tests continue passing to detect regressions.
- Problem Statement Writer Agent: The Problem Statement Writer Agent synthesizes self-contained, solution-leakage-free task descriptions from pull-request metadata and executable tests.Grounding descriptions in tests improves alignment with actual validation requirements and addresses the retrospective or incomplete nature of many pull-request descriptions.
- Dataset Statistics: Table 1 reports dataset statistics using means and P50, P75, and P95 percentiles for code modifications and test-case distributions.These statistics characterize both task changes and validation-test coverage.
2.2. Data Processing
Scale-SWE curates verified software-engineering instances from real repositories using filtering, automated validation, and expert auditing. The resulting dataset combines broad repository coverage with realistic task complexity and diverse bug types.
- Data processing: The curation pipeline filters merged pull requests, sanitizes environments, and retains instances only when P2P and F2P tests satisfy prescribed outcomes.Quality control combines automated test-based filtering with manual expert auditing.
- Data processing: An audit of 100 randomly sampled instances confirmed 94 valid environments, test suites, and problem statements.The audit used four senior Ph.D. students and a cross-validation protocol.
- Dataset construction: 100,000 verified instances were constructed from 5,200 real repositories, increasing repository count by 50% over SWE-rebench’s 3,500.The dataset is described as the largest verified SWE benchmark and is intended to reflect real-world task variety.
- Dataset statistics: The median instance modifies at least 3 files and adds 43 lines, while containing over 200 P2P tests on average and 5.69 F2P tests.These statistics indicate substantial task complexity and validation coverage.
- Bug diversity: Scale-SWE presents a balanced distribution across ten bug categories, contrasting with datasets showing stronger category bias or limited repository variety.The taxonomy includes categories ranging from constructor errors to security flaws.
3. Experiments
Experiments use OpenHands with Qwen3-30B-A3B-Instruct and evaluate resolved solutions on SWE-bench Verified. Scale-SWE training data improves performance over the base model and alternative data sources while representing complex debugging workloads.
- Experimental framework: OpenHands provides the unified sandboxed framework for editing files, executing shell commands, and browsing the web during agent experiments.The framework was selected for robust and reproducible SWE-bench baselines.
- Evaluation setup: SWE-bench Verified contains 500 human-curated Python issues, and Resolved Rate measures the proportion receiving a correct solution.Inference context was extended to 262,144 tokens despite 131,072-token training sequences.
- Experiment results: 42.0% absolute improvement raises the Qwen3-30B-A3B-Instruct pass rate from 22.0% to 64.0% on SWE-bench Verified.Scale-SWE Agent also exceeds listed same-size competitors and KAT-Dev-32B.
- Data comparison: Scale-SWE significantly outperforms SWE-Gym and SWE-smith under an identical distillation and supervised-fine-tuning pipeline.SWE-smith has more instances than SWE-Gym but produces slightly inferior performance.
- Task complexity: Scale-SWE tasks require more interaction turns for repository exploration and iterative debugging, indicating high task complexity.The interaction-turn and token-count distributions are reported in Figure 4.
4. Related Work
Related work spans increasingly specialized benchmarks, SWE datasets, and agent scaffolds for evaluating and improving repository-level software engineering. Existing dataset efforts differ in their reliance on synthetic or real-world sources, while Table 4 frames model comparisons under a shared distillation pipeline.
- SWE Benchmark: SWE benchmarks now assess multimodal, multilingual, long-horizon, whole-repository, scientific, and other specialized software-engineering capabilities.
- SWE Datasets: Repository-level SWE datasets pursue scale through synthetic generation or other approaches to complex coding-task construction.
- SWE Models and Agents: Specialized SWE models and agent frameworks improve coding performance and streamline interaction with development environments.
- SWE Models and Agents: SFT comparisons on SWE-bench Verified use the same distillation pipeline for all models, supporting a controlled comparison.
5. Conclusion
Scale-SWE automates large-scale construction of high-quality SWE data and executable environments through sandboxed specialized agents. The resulting dataset supports trajectory distillation and fine-tuning, producing substantial SWE-bench improvements and an open resource for future agent development.
- Scale-SWE automates construction of large-scale, high-quality software-engineering data with executable environments through a sandboxed multi-agent framework.
- 100,000 instances from six million real-world pull requests surpass existing datasets in scale and repository diversity.
- Fine-tuning Qwen3-30B-A3B-Instruct on distilled trajectories increases SWE-Bench-Verified resolve rate from 22% to 64%.
- Scale-SWE provides an open-access resource intended to support development of more capable LLM-based software-engineering agents.
A.1. Scale-SWE workflow Overview.
The Scale-SWE workflow filters a large repository and pull-request pool before converting selected pull requests into formal SWE tasks. Specialized agents handle environment construction, test creation, and task-statement writing inside a sandboxed orchestration.
- The workflow starts from 23k repositories and 6M pull requests, then uses an LLM-as-a-judge to filter for quality and relevance.
- The pipeline selects 1M pull requests for transformation into formal software-engineering task instances.
- Sandboxed specialized agents build environments, create tests, and write task statements for the selected pull requests.
B. Scale-SWE Task Instance Structure
Scale-SWE task instances follow the SWE-bench structure while adding generated fail-to-pass tests for mined instances lacking developer-written tests. The schema records repository identity, execution environment, patches, commits, and regression-oriented test sets.
- Scale-SWE adapts the SWE-bench structure by adding generated fail-to-pass scripts when original developer-written tests are unavailable.
- Each instance records identifiers, repository ownership, repository name, programming language, and working-directory information.
- The schema includes a Docker image, ground-truth patch, pull-request commit, and GitHub repository URL for reproducible execution and provenance.
- FAIL_TO_PASS tests distinguish buggy from fixed behavior, while PASS_TO_PASS tests provide regression coverage across both versions.
C. Implementation Details
The SFT hyperparameters are documented in Table 5, which presents the key settings for this phase.
- Table 5 details the hyperparameters used during supervised fine-tuning.
- The implementation details direct readers to Table 5 for the SFT configuration.
- The table is identified as covering key hyperparameters in the SFT phase.
D. Anti-hack Strategy
The anti-hack strategy sanitizes the repository immediately after initialization to remove future Git history and prevent access to ground-truth solutions.
- The sanitization script runs immediately after environment initialization to prevent access to ground-truth solutions through Git history.
- The process hard-resets the repository and removes remote references, tags, and internal Git metadata.
- Branch cleanup deletes existing references while preserving a designated working branch.
E. Prompts in Scale-SWE workflow
The Scale-SWE workflow uses specialized prompts to filter repositories and pull requests, configure environments, generate verified tests, and write problem statements for software-engineering benchmarks.
- Repository filtering: Repository prompts reject deep-learning projects and API wrappers while selecting projects with substantial internal software logic.
- Pull-request filtering: Pull-request prompts classify changes as suitable when they contain substantive code-logic modifications rather than documentation, trivial, or generated-file changes.
- Environment setup: The environment-builder prompt installs dependencies in an existing conda environment so repository tests can run successfully.
- Unit-test creation: The unit-test creator generates 2 to 10 pytest functions and verifies that they fail before the pull request and pass afterward.
- Unit-test creation: Generated tests must exercise public APIs rather than directly calling newly added functions to reduce overfitting.