Source-linked AI summary
Toward Training Superintelligent Software Agents through Self-Play SWE-RL
Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, Sida Wang
TL;DR
Software-agent RL remains dependent on human-curated issues, tests, and development traces, limiting training from autonomous software experience. SSR instead trains one LLM through self-play on sandboxed repositories, where it injects and repairs formally specified bugs. SSR shows steady self-improvement and outperforms human-data baselines across SWE-bench Verified and SWE-Bench Pro, although the authors describe the results as early and note practical safety and complexity boundaries.
Problem
Current software-agent training relies heavily on human-curated data and environments, while existing synthetic bug generation retains human-data assumptions and static pipelines.
Method
SSR uses one LLM policy in bug-injection and bug-solving roles to generate and repair increasingly complex, formally specified bugs using only sandboxed repositories with source code and dependencies.
Results
SSR steadily self-improves and outperforms human-data baselines across SWE-bench Verified and SWE-Bench Pro throughout training.
Takeaways & Limitations
Self-play SWE-RL suggests a path for agents to autonomously gather learning experiences from real-world software repositories toward broader software understanding and creation.
Takeaways & Limitations
Generating bugs through code removal can produce unnatural or initially overly difficult tasks and may require solvers to reconstruct code from scratch.
Abstract
from arXiv · showhide
While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.g., GitHub issues and pull requests) and environments (e.g., pass-to-pass and fail-to-pass tests) heavily depend on human knowledge or curation, posing a fundamental barrier to superintelligence. In this paper, we present Self-play SWE-RL (SSR), a first step toward training paradigms for superintelligent software agents. Our approach takes minimal data assumptions, only requiring access to sandboxed repositories with source code and installed dependencies, with no need for human-labeled issues or tests. Grounded in these real-world codebases, a single LLM agent is trained via reinforcement learning in a self-play setting to iteratively inject and repair software bugs of increasing complexity, with each bug formally specified by a test patch rather than a natural language issue description. On the SWE-bench Verified and SWE-Bench Pro benchmarks, SSR achieves notable self-improvement (+10.4 and +7.8 points, respectively) and consistently outperforms the human-data baseline over the entire training trajectory, despite being evaluated on natural language issues absent from self-play. Our results, albeit early, suggest a path where agents autonomously gather extensive learning experiences from real-world software repositories, ultimately enabling superintelligent systems that exceed human capabilities in understanding how systems are constructed, solving novel challenges, and autonomously creating new software from scratch.
1. Introduction
Existing software-agent RL relies heavily on human-curated data and environments, limiting autonomous discovery. SSR addresses this by using self-play on raw repositories to generate and solve formalized bugs, and reports stronger performance than human-data training.
- Motivation: SSR targets the dependence of software-agent training on human-authored issues, tests, and development traces.Existing agents primarily replay and refine human software-development traces, while synthetic bug methods retain stronger data assumptions and static generation pipelines.
- Approach: SSR requires only sandboxed repositories with source code and dependencies, without existing tests, test runners, issue descriptions, or language-specific infrastructure.In practice, each input is a pre-built Docker image.
- Approach: The same LLM policy alternates between bug injection and bug solving, creating increasingly challenging training experiences through self-play.The injector constructs validated artifacts including code patches, tests, parsers, and a test-weakening patch; the solver receives a formal specification derived from the reversed weakening patch.
- Results: SSR consistently surpasses the human-data baseline on SWE-bench Verified and SWE-Bench Pro throughout the training trajectory.The baseline uses the same hyperparameters and environment images but receives human-authored or human-curated issue descriptions, test suites, and test commands.
2. Self-Play SWE-RL
Self-play SWE-RL trains one LLM policy to generate and repair increasingly complex software bugs using sandboxed repositories, formal test artifacts, and reinforcement learning. The approach validates bug artifacts for consistency, expands training with higher-order bugs, and uses solver feedback to improve bug generation.
- Self-play SWE-RL: SSR divides one LLM policy into bug-injection and bug-solving roles that share a containerized environment and tools while receiving different task specifications.The injector creates challenges; the solver repairs them through interaction with the buggy codebase.
- Self-play SWE-RL: SSR assumes only Docker images containing source repositories with installed dependencies, without requiring existing tests, test commands, parsers, or programming-language knowledge.This design minimizes prior knowledge about the codebase.
- Bug artifacts: Each bug artifact combines an inducing code patch, test script and files, a test parser, and a weakening patch whose reversal specifies the solver’s required behavior.The weakening patch hides the bug from existing tests, while the reversed patch serves as the oracle test specification.
- Reward design: The injector learns through reinforcement learning from consistency and solver feedback, with its reward based on solver solve rate and a penalty for degenerate rates of 0 or 1.The penalty magnitude is controlled by α, set to 0.8 in the experiments.
- Complex bug generation: Bug injection uses code-hunk removal and historical-change reversion, while higher-order bugs derive from failed solver attempts to produce new challenges across the agent’s coding capabilities.Removal-based bugs can require reconstructing large code regions and may be less natural than real repository bugs, motivating higher-order generation.
- Consistency validation: Valid artifacts undergo checks for test existence and coverage, parser and script validity, bug scope and failure induction, test weakening, and file-level mutation necessity.Passing artifacts are reformatted into SWE-bench-like instances with pass-to-pass and fail-to-pass test specifications.
3. Evaluation
SSR steadily improves software-engineering performance from raw environment images and consistently outperforms baseline RL across both benchmarks. Ablations attribute this advantage to online self-play and challenging, history-informed bug generation, while solver feedback adds little.
- 3.2. Baseline Comparison: SSR steadily self-improves without task-specific training data and consistently outperforms baseline RL across both benchmarks throughout training.Both methods use identical environment images, but SSR relies only on self-play while baseline RL receives natural-language issues, tests, and evaluation scripts.
- 3.3. Ablation of Self-Play Components: Self-play performs best because joint bug generation and repair continually expands the training signal and exposes new failure modes.Injection-only lacks repair learning, whereas repair-only lacks the evolving task distribution produced by self-play.
- 3.4. Ablation of Bug Injection Methods: SSR remains effective across direct-injection, removal-only, and removal + history strategies, demonstrating robustness to injection-method choice.The three variants differ in whether they use naive injection, code removal, or code removal combined with historical-change reversion.
- 3. Evaluation: The evaluation averages resolve rate over 1231 tasks spanning 500 SWE-bench Verified and 731 SWE-Bench Pro problems.The benchmark evaluation uses one attempt per problem without parallel test-time scaling or ranking.
- 3.4. Ablation of Bug Injection Methods: Removal + history achieves the best overall performance, while direct injection performs worst because superficial one-line bugs provide little training signal.Removal-only creates stronger tasks requiring reconstruction of missing functionality; history-aware removal adds realistic and diverse bug patterns.
- 3.5. Reward Ablation: Solver-feedback rewards provide only a slight, largely negligible advantage over binary consistency rewards because solve-rate feedback is weak and noisy.Even without solver feedback, the continuously updated joint policy generates an evolving curriculum reflecting the agent’s current capabilities.
4. Related Work
Related work develops software-engineering agents through agentic scaffolds, reinforcement learning, and increasingly executable real or synthetic data. These approaches still commonly rely on curated issues, tests, parsers, or generated task pipelines, motivating SSR’s raw-codebase self-play setting.
- Agent scaffolding for software engineering: Agentic scaffolds use LLMs to make software-engineering decisions through tool-mediated interaction with sandboxed environments.This work describes agentic scaffolds as one of two major scaffold families for improving issue-solving ability on repository-level tasks.
- Training software agents via reinforcement learning: Reinforcement-learning methods improve software-engineering competence using verifiable rewards and software-evolution data, with several open models emphasizing agentic intelligence.Examples include SWE-RL and DeepSWE, alongside newer open LLMs trained extensively on software-engineering data and environments.
- Data and environments for training software agents: Training-data research combines real-data collection with synthetic generation, producing executable environments and tasks through curation, perturbation, commit translation, or automated extraction.Examples include SelfAPR, SWE-Gym’s 2.4k curated tasks, R2E-Gym’s 8k-plus generated tasks, and SWE-rebench’s 21k-plus extracted Python tasks.
5. Conclusion
SSR is presented as an early self-play approach for training software agents with minimal human-curated data. Its benchmark results show steady self-improvement and performance above human-data baselines, suggesting a path toward agents that autonomously learn from real-world repositories.
- SSR outperforms human-data baselines throughout training on SWE-bench Verified and SWE-Bench Pro while showing steady self-improvement.
- The approach aims to let agents autonomously gather extensive experience from real-world repositories and eventually exceed human capabilities in software understanding, problem solving, and creation.
Impact Statement
Self-play SWE-RL may reduce reliance on curated data for software-engineering training and support debugging when labeled data is scarce or outdated. However, generating and weakening tests creates misuse risks that require careful safeguards.
- Self-play bug generation may reduce data-collection costs and improve training scalability for software-engineering tasks.
- The approach could support automated debugging and maintenance where labeled data is scarce or outdated.
- Intentional bug generation and test weakening can enable vulnerable or misleading artifacts, requiring sandboxing, responsible deployment, and ethical safeguards.
A. Analysis of the Challenger-Solver Game
The challenger-solver formulation differs from standard game self-play because a sufficiently intelligent challenger can adopt dominant strategies that stop the solver from learning useful behavior. The analysis motivates grounding and limiting self-play to address these failure modes.
- A sufficiently intelligent challenger can use dominant strategies that make the solver learn nothing useful and halt self-play progress.
- The analysis identifies both a powerful-action-space challenger and a tunnel-vision challenger as distinct sources of stalled learning.
- Strong grounding and limited self-play are discussed as practical mitigations for challenger-driven failure modes.
A.1. The Challenger Has an Optimal Target Solve Rate
The challenger’s expected reward has an interior optimum target solve rate rather than favoring either zero or perfect solvability. With the analyzed reward, the optimum is around 0.2, and smoothing makes it behave similarly to a Beta reward.
- The challenger reward has a unique optimal target solve rate p* in (0, 1), avoiding both trivially solved and impossible bugs.
- For the analyzed reward, the expected reward has a unique maximum around target solve probability 0.2.
- Figure 10 shows that expected reward smooths the original discontinuous reward and makes it similar to a Beta reward with suitable parameters.
A.2. The Challenger Strategy
The challenger can exploit unrestricted problem-setting to make self-play degenerate, so the analysis proposes grounding and constraining its behavior. These safeguards aim to preserve diverse, informative challenges without allowing reward-maximizing strategies to halt learning.
- The dominant challenger: The challenger’s dominant strategy arises when it has enough freedom to set problems whose solve probability determines its reward regardless of solver behavior.The analysis defines this as a challenger strategy that achieves optimal reward by selecting a problem with probability p* of being solved.
- The dominant challenger: Unrestricted challengers can win by injecting fail-randomly tests or obfuscating code, leaving the solver unable or unmotivated to improve.Theoretical examples include deterministic pseudo-random tests and codebases made unintelligible to the solver.
- The tunnel-vision challenger: A tunnel-vision challenger can tune one difficulty parameter until the solver’s learning saturates without exposing it to diverse challenges.Examples include increasing operand length, bug obfuscation, or chaining medium-difficulty tasks.
- Mitigations: Self-play should improve technical and code-world skills, but not natural-language communication without continued grounding in current human language.The text connects this boundary to cooperation settings where self-play policies may diverge from human behavior or language.
- Mitigations: The proposed mitigation is to ground challenges in large, diverse real-world data and prevent the challenger from drifting far from its initial instruction-based strategy.These constraints support diversity, consistency checks, and calibration of question difficulty.
C. Experimental Details
The experimental implementation uses the CWM-RL infrastructure and constructs training bugs through removal-oriented and history-aware injection procedures. These procedures require validated tests, parsers, patches, and coverage constraints so generated bugs remain runnable and diagnostically useful.
- Training details: SSR is implemented on async CWM-RL with large-scale GPU training, while retaining CWM’s tool scaffold and environment images.The implementation uses Bash, a search-replace editor, and the same environment images as CWM.
- Removal-oriented bug-injection: Removal-oriented injection deletes code files or hunks, reruns tests to confirm broken functionality, and creates separate code and test-weakening patches.The process requires parsing test logs and ensuring the weakening patch changes only test files.
- Validation requirements: Injection requires enough initially passing tests and coverage of every modified code file, while excluding flaky or environment-sensitive tests.The workflow also requires test execution within a bounded runtime and forbids orphan modified files.
- Bug artifacts: Each bug package includes selected test files, a test script, a parser, a bug patch, and a test patch for reproducible validation and submission.The parser maps executed tests to passed or failed outcomes in JSON format.
- History-aware bug-injection: History-aware injection selectively reverts historical fixes or improvements across files and applies only minimal compatibility fixes needed to keep the repository runnable.The resulting bugs are intended as training data for bug-fixing systems while preserving historical defects.
E.1. Theoretical Analysis
The theoretical analysis studies the challenger’s reward design and optimal play in the challenger–solver game. It identifies dominant strategies that can stop self-play and motivates practical constraints adopted by the method.
- Theoretical analysis: The analysis examines Equation (1) and the theoretical optimal play of the challenger–solver game from the challenger’s perspective.This frames the bug-injector as the central source of self-play dynamics and failure modes.
- Theoretical analysis: Dominant challenger strategies can stop self-play from progressing by exploiting the freedom to construct pathological challenges.The analysis treats these strategies as behaviors that must be avoided rather than as useful learning targets.
- Theoretical analysis: The analysis uses these results to clarify expected self-play behavior and motivate practical mitigations, some of which the work adopts.The stated purpose is to identify both what self-play can produce and which challenger behaviors should be constrained.
E.2. Limitations
The paper identifies three limitations of SSR: complete test oracles may invite reward hacking, unit tests cover only part of software engineering, and experiments use one model configuration for both roles.
- No hidden oracles: Providing complete tests in the task specification may encourage overfitting to those tests rather than genuine bug-fixing, motivating public–private test splits.The paper states that this reward-hacking behavior was not observed in the current work.
- Unit tests for verification: Using unit tests as the sole verification oracle covers only a subset of real-world software engineering activities.The paper suggests higher-level goals as a potentially more scalable correctness signal.
- Lack of model variants and role separation: Experiments use one model configuration for both roles, leaving the effects of model capacity, architecture, and role separation for future study.The paper specifically proposes larger Mixture-of-Experts models and separate policies.
E.3. Unsuccessful Attempts
The authors report several unsuccessful directions and scaling limitations, including unreliable natural-language issue synthesis, repository-specialized training that did not outperform broader training, and instability at scale. They identify future work on controlling bug distributions, handling complex multi-step tasks, and making long-horizon training more efficient.
- Synthesizing natural language issues: Natural-language issue synthesis was unreliable: generated issues copied test patches, became logically incoherent, and collapsed to identical patterns.The authors attribute these failures to limited natural-language capabilities and opaque reward signals that did not promote quality or diversity.
- Repository-specialized training: Training on only 23 repository images did not outperform training on more non-overlapping repositories.The experiment selected commits before evaluation instances to prevent leakage; the authors suggest the limited number of repositories may explain the result.
- Stable training at scale: Training instability prevented SSR from scaling further and manifested as gibberish outputs.Possible contributors include hyperparameters, long-horizon rollouts, pretraining dynamics, and fundamental properties of self-play learning.
- Distribution control with seeding: Current bug injection lacks explicit distribution control, risking duplicate bugs or sampling bias within a repository.The authors propose seeding bug injection with target snippets or files to improve diversity.
- Synthesizing complex multi-step software tasks: SSR handles repository-level bug fixing, but complex workflows such as version migrations or building software stacks from scratch require new scaffolding.Proposed directions include multi-context rollouts across interdependent sessions and automated context compaction.
- Efficient training paradigms for long-horizon software agents: Outcome-based RL is inefficient for long-horizon projects because sparse terminal rewards provide little signal when most trajectories fail.Production-grade software projects can span months, thousands of interdependent decisions, and validation beyond unit tests.