Source-linked AI summary
SWE-Universe: Scale Real-World Verifiable Environments to Millions
Mouxiang Chen, Lei Zhang, Yunlong Feng, Xuwu Wang, Wenting Zhao, Ruisheng Cao, Jiaxi Yang, Jiawei Chen, Mingze Li, Zeyao Ma, Hao Ge, Zongmeng Zhang, Zeyu Cui, Dayiheng Liu, Jingren Zhou, Jianling Sun, Junyang Lin, Binyuan Hui
TL;DR
Real-world SWE environments are difficult to scale because existing construction pipelines face low yield, weak verifiers, and high cost. SWE-Universe uses an autonomous building agent with iterative self-verification, in-loop hacking detection, and an efficient custom-trained model to construct multilingual environments at scale. It produces 807,693 instances and, through agentic training and reinforcement learning, enables Qwen3-Max-Thinking to achieve 75.3% on SWE-Bench Verified.
Problem
Scaling reliable, diverse, multilingual SWE environments remains difficult because construction pipelines face low production yield, weak verifiers, and prohibitive cost.
Method
SWE-Universe uses an autonomous building agent that synthesizes executable environments and verifiers, iteratively validates them on buggy and fixed states, detects hacking, and relies on an efficient custom-trained model.
Results
807,693 multilingual, verifiable instances were constructed, and Qwen3-Max-Thinking achieved 75.3% on SWE-Bench Verified after agentic training and reinforcement learning.
Takeaways & Limitations
The resulting environments provide training data and a reward signal for developing more capable and versatile coding agents for real-world applications.
Takeaways & Limitations
The dataset still contains ambiguous or incomplete task descriptions, mismatched Docker environments, and tests misaligned with descriptions, while the mid-training benefit is presented as a hypothesis.
Abstract
from arXiv · showhide
We propose SWE-Universe, a scalable and efficient framework for automatically constructing real-world software engineering (SWE) verifiable environments from GitHub pull requests (PRs). To overcome the prevalent challenges of automatic building, such as low production yield, weak verifiers, and prohibitive cost, our framework utilizes a building agent powered by an efficient custom-trained model. This agent employs iterative self-verification and in-loop hacking detection to ensure the reliable generation of high-fidelity, verifiable tasks. Using this method, we scale the number of real-world multilingual SWE environments to a million scale (807,693). We demonstrate the profound value of our environments through large-scale agentic mid-training and reinforcement learning. Finally, we applied this technique to Qwen3-Max-Thinking and achieved a score of 75.3% on SWE-Bench Verified. Our work provides both a critical resource and a robust methodology to advance the next generation of coding agents.
1 Introduction
SWE-Universe addresses the difficulty of scaling reliable, multilingual real-world SWE environments by combining an autonomous building agent with self-verification and hacking detection. The framework produces 807,693 verifiable training instances and supports agentic training that reaches 75.3% on SWE-Bench Verified.
- Motivation: Scaling real-world verifiable environments requires addressing low production yield, weak verifiers, and prohibitive cost.These challenges arise from heterogeneous repositories, superficial verification scripts, and expensive repository-specific reasoning.
- Framework: SWE-Universe uses an autonomous building agent to synthesize a self-contained executable environment and verifier for each pull request.The agent repeatedly tests the verifier on buggy and fixed repository states, improving build success from 82.6% to 94% on a held-out set.
- Framework: An in-loop hacking detector rejects superficial verifiers, while a custom-trained efficient model reduces build latency and cost.The detector targets scripts that pass through shallow checks rather than executing the intended code.
- Scale: 807,693 multilingual, verifiable training instances were constructed from over 52,000 unique GitHub repositories.The dataset is presented as the largest and most diverse collection of real-world software engineering tasks with executable verification.
- Validation: Large-scale agentic training shows that the environments improve benchmark performance and provide a reward signal for reinforcement learning.Applied to Qwen3-Max-Thinking, the approach achieves 75.3% on SWE-Bench Verified.
2 Methodology: Scalability and Reliability
The methodology separates PR tests from fixes, builds universal executable verifiers, and validates them across buggy and resolved states. Hacking detection rejects static checks that do not establish runtime correctness, while quality analysis identifies remaining dataset issues.
- Data preparation: The pipeline separates each pull request into a test patch and a fix patch before environment construction.Pull requests without a discernible test component are discarded.
- Agent-based Environment Building: The agent generates evaluation.sh to distinguish buggy and fixed repository states through a universal bash interface and return code.It may invoke existing unit tests or author a custom test when the supplied tests lack a straightforward execution entry point.
- Agent-based Environment Building: The universal verifier interface decouples verification logic from language-specific conventions, supporting generalization across projects and ecosystems.The design avoids hardcoding workflows such as pytest or cargo run.
- Iterative Validation: A verifier passes iterative validation only when it exits non-zero on the buggy state and zero on the fixed state.Failure feedback is returned to the agent so it can revise the build procedure.
- In-loop Hacking Detection: Hacking detection rejects scripts that use grep or similar string matching instead of executing builds or tests.Executable tests in the JavaScript and Python cases are accepted, whereas Scala static pattern matching is rejected because expected code patterns may coexist with logical errors.
- Quality Analysis: The quality-judge agent reaches 78.72% accuracy, while the dataset matches SWE-Rebench quality with 38× more instances.Observed issues include ambiguous descriptions, mismatched Docker environments, and tests misaligned with task descriptions.
3 Efficient Building and Benchmarking
SWE-Universe improves automated environment building through an efficient custom-trained model and evaluates reliability across diverse pull requests using hacked and non-hacked success metrics.
- Benchmark design: The benchmark contains 320 randomly sampled GitHub pull requests across eight language categories.The categories include Python, JavaScript/TypeScript, Go, Java, Rust, C/C++, C#, and Others.
- Metrics: Success Rate (w/o Hack) requires functional correctness and hacking-detector passage, whereas Success Rate (w/ Hack) also counts hacked verifiers.These metrics distinguish valid verification from scripts that merely distinguish repository states.
- Reliability: 4.06% is Qwen-Next-80A3’s gap between true success and success including hacked verifiers, the smallest gap among top performers.Claude-Opus-4.5 shows a gap exceeding 7%.
- Reliability: Qwen-Next-80A3 performs more consistently across languages, while C/C++ is the most challenging category for all agents.The results associate task-specific fine-tuning with reduced superficial verification behavior.
4 Scaling Environments to Millions
The pipeline scales from a large GitHub pull-request corpus to hundreds of thousands of executable, multilingual environments using distributed infrastructure and an efficient builder model.
- Data curation: 33.3 million pull requests from 2021–2025 were filtered using constraints on changes, line counts, test patches, and task quality.The filters target overly complex, noisy, or unverifiable candidates.
- Infrastructure: MEGAFLOW dispatches each environment-building task as an independent job to a dedicated Alibaba Cloud ECS instance.The distributed execution system supports large numbers of long-running agentic jobs.
- Massive production: 75.9% non-hacked success produced 717,122 executable, high-fidelity environments from filtered candidates.The environments primarily assess resolution of real-world software issues.
- Massive production: 90,571 additional environments were synthesized from 2025 pull requests without linked issues, using titles and descriptions as problem statements.This expands coverage beyond issue-linked pull requests.
- Dataset statistics: 807,693 instances span 52,960 unique repositories, with Python and JavaScript/TypeScript comprising the largest language shares.Go repositories have the highest average number of instances per repository at 21.80.
5 Evaluation: Large-scale Agentic Training
Large-scale agentic mid-training and reinforcement learning on SWE-Universe environments improve performance across standard and multilingual coding benchmarks, culminating in a 75.3% SWE-Bench Verified result for Qwen3-Max-Thinking.
- Mid-training: Mid-training produces a positive scaling trend on SWE-Bench Verified and SWE-Bench Multilingual.Checkpoints were evaluated during training to measure transfer from the real-world instances to standard benchmarks.
- Mid-training: SWE-Bench Multilingual increases from approximately 31% to over 46%, a gain of over 15 percentage points.The passage attributes the stronger multilingual improvement to the dataset’s linguistic diversity.
- Agentic reinforcement learning: Qwen3-30B-A3B improves on SWE-Bench Multilingual from approximately 32% to a 42.0% peak with agentic reinforcement learning.The reported absolute gain is 10 points.
- Applied model: Qwen3-Max-Thinking achieves 75.3% on SWE-Bench Verified after applying the methodology.This result is presented as validation of the production-level data-generation pipeline.
6 Related Work
Related work spans synthetic SWE task generation, executable environment setup, and automated code verification; SWE-Universe focuses on executable environments for real-world software problems from public repositories.
- Agentic and synthetic generation: SWE-smith, SWE-Flow, and related studies generate software engineering tasks synthetically rather than relying exclusively on historical issues.Their approaches include injecting bugs, using test documentation, and generating complex bugs from scratch.
- Agentic and synthetic generation: SWE-Universe builds executable environments for real-world software engineering problems derived directly from public repositories.This distinguishes it from synthetic task-generation approaches.
- Real-world environment setup: Some environment-setup methods create executable environments without providing verifiers for specific software issues.Other work provides end-to-end verification for real software issues.
- Building code verifiers: Traditional automated test-generation methods often have limited coverage and poor readability and commonly target regression or implicit oracles.The cited families include probability-based, constraint-based, and search-based approaches.
7 Conclusion
SWE-Universe addresses bottlenecks in real-world SWE environment generation with an autonomous building agent and iterative safeguards. The resulting multilingual dataset supports large-scale training and reaches 75.3% on SWE-Bench Verified.
- SWE-Universe targets low yield, inconsistent quality, and prohibitive cost in generating real-world software engineering environments.
- The autonomous building agent uses a custom-trained model, iterative self-verification, and in-loop hacking detection.
- Over 800,000 executable, multilingual tasks were constructed as the largest dataset of its kind.
- Large-scale agentic training showed the dataset provides signals for both supervised learning and reinforcement learning.
- 75.3% on SWE-Bench Verified was achieved by Qwen3-Max-Thinking using this technique.