Source-linked AI summary
SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, Sida I. Wang
TL;DR
Existing reinforcement-learning methods have shown limited effectiveness for software engineering, motivating methods that can scale reasoning to real-world software tasks. SWE-RL trains an LLM with rule-based rewards on software evolution data, and Llama3-SWE-RL-70B reaches 41.0% on SWE-bench Verified while improving out-of-domain reasoning performance. The results support reinforcement learning on open software engineering data as a direction for improving LLM reasoning.
Problem
Existing rule-based reinforcement learning has limited effectiveness for software engineering and has mainly been studied on competitive coding and mathematics tasks.
Method
SWE-RL trains an LLM to solve real-world software issues using software evolution data and rule-based rewards based on similarity to oracle patches.
Results
Llama3-SWE-RL-70B solves 41.0% of SWE-bench Verified issues, achieves state-of-the-art performance among medium-sized models, and improves on out-of-domain tasks while the SFT baseline declines.
Takeaways & Limitations
Reinforcement learning applied solely to software engineering issues can improve generalized reasoning abilities across code, mathematics, and language tasks.
Takeaways & Limitations
The reward compares sequence similarity rather than semantic equivalence, potentially limiting exploration of functionally equivalent solutions.
Abstract
from arXiv · showhide
The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer's reasoning processes and solutions by learning from extensive open-source software evolution data -- the record of a software's entire lifecycle, including its code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified -- a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.
1 Introduction
SWE-RL applies rule-based reinforcement learning to software evolution data for real-world software engineering tasks. The resulting Llama3-SWE-RL-70B achieves strong SWE-bench performance and improves reasoning on out-of-domain tasks.
- Conclusion: The paper introduces SWE-RL as a new direction for enhancing LLM reasoning through reinforcement learning on open software engineering data.Its contribution is positioned against prior reinforcement-learning work focused mainly on coding and mathematics.
- Motivation and approach: SWE-RL is the first approach described here to apply rule-based reinforcement learning to software engineering using software evolution data.The data records software lifecycles, including code snapshots, changes, pull requests, and issues.
- Motivation and approach: The method curates GitHub pull requests into examples containing issue descriptions, code context, and oracle patches for policy-generated repairs.The policy produces code changes through reasoning, which are converted to a consistent patch format for reward calculation.
- Results: Llama3-SWE-RL-70B improves on out-of-domain tasks, while the supervised-finetuning baseline produces decreased performance on those tasks.The reported out-of-domain areas include function-level coding, library use, code reasoning, mathematics, and general language understanding.
- Results: 41.0% of SWE-bench Verified issues are solved by Llama3-SWE-RL-70B, establishing the reported best result among medium-sized models below 100B parameters.The evaluation uses Agentless Mini, a pipeline-based scaffold built upon Agentless.
2 SWE-RL
SWE-RL trains a policy LLM to reason over issue descriptions and code context before producing repair edits, using software-evolution data and rule-based rewards. The resulting model also exhibits emergent reasoning behaviors and transfers them to out-of-domain tasks.
- SWE-RL training data: 273k high-quality pull-request seeds provide issue descriptions, code context, and oracle patches for reinforcement-learning training.Seeds are filtered for linked bug-fixing issues and programming-file changes.
- SWE-RL training data: The policy LLM receives an issue and code context, then generates search/replace edits as the sole reinforcement-learning subtask.During inference, the model can support additional tasks such as file-level localization and test generation.
- Training and inference scope: Complete-file context implicitly trains both fault diagnosis and repair generation, even though training directly targets repair edits.Evaluation additionally requires localization, reproduction-test generation, and regression-test selection.
- Emergent reasoning capabilities: SWE-RL produces emergent self-reflection, alternative exploration, and divide-and-conquer reasoning in issue-solving, function implementation, mathematics, and other tasks.The model allocates more thinking time to reconsider initial assumptions, and improvements extend to library use, code reasoning, and general language understanding.
3 Evaluation
Evaluation shows that SWE-RL improves real-world issue solving, scales with additional repair and test samples, generalizes across five out-of-domain tasks, and benefits from continuous rewards.
- 3.2 Main results: 41.0% pass@1 on SWE-bench Verified makes Llama3-SWE-RL-70B state-of-the-art among language models below 100B parameters.It uses publicly available data and does not rely on proprietary LLMs in the pipeline.
- 3.3 Baseline comparison: SWE-RL improves repair capabilities over both the base Llama-3.3 model and the SFT baseline, despite slightly lower format accuracy than SFT.The repair-only comparison provides oracle localized files and uses greedy decoding.
- 3.4 Scaling analysis with more samples: 33.6 to 40.0: increasing repair samples from 20 to 160 substantially improves SWE-bench performance, while gains plateau from 160 to 500 samples.With 500 repair samples, increasing reproduction tests improves scores from 38.8 to 41.0 by 20 tests, with no difference between 20 and 30 tests.
- 3.5 Generalizability of RL: Across five out-of-domain tasks, Llama3-SWE-RL-70B outperforms both its base model and the SFT baseline, while SFT underperforms the original model on average.The tasks cover function coding, library use, code reasoning, mathematics, and general language understanding.
- 3.5 Generalizability of RL: > 0.8 percentage points on MMLU, 3 points on CRUXEval, and > 3 points on the full MATH dataset are individually significant thresholds; together, the improvements reach significance at the 0.05 level.Small absolute gains on individual HumanEval or CRUXEval results are not necessarily significant by themselves.
- 3.6 Reward ablation: Continuous rewards improve repair performance more effectively than discrete rewards because they capture partial correctness and incremental improvements.Discrete rewards grow more slowly and remain approximately zero at training completion because exact oracle-patch matches are difficult.
4 Related work
Related work applies LLMs to diverse software-engineering tasks, while open models remain comparatively weak and the best training approach for real-world software engineering is unclear.
- 4 Related work: Research has explored LLMs for library-level and complex code generation, real-world bug and issue resolution, and software testing.SWE-bench, a benchmark for real-world software issues, has become a focal point of research.
- 4 Related work: Open models typically perform poorly on real-world software-engineering tasks, and the most effective training approach remains unclear.Recent work has begun training open LLMs specifically for software-engineering benchmarks.
5 Conclusion
SWE-RL applies reinforcement learning to software engineering using software evolution data and rule-based rewards, achieving strong issue-solving performance while developing generalized reasoning skills. The approach remains constrained by sequence-based rewards and structural limitations in its scaffold.
- SWE-RL is the first RL approach using software evolution data and rule-based rewards to improve LLMs on software engineering tasks.
- 41.0% solve rate on SWE-bench Verified makes Llama3-SWE-RL-70B state-of-the-art among medium-sized models and comparable to proprietary LLMs such as GPT-4o.
- Llama3-SWE-RL-70B improves performance on out-of-domain code reasoning, mathematics, and general language understanding tasks.
- The reward compares predicted and oracle patch sequence similarity rather than semantic equivalence, potentially discouraging functionally equivalent alternatives.
- Agentless Mini’s simplified localization and staged pipeline limit contextual understanding, interaction feedback, and holistic problem consideration.
NeurIPS Paper Checklist
The checklist reports that the paper states its contributions, describes empirical evaluations and reproducibility materials, and discusses limitations. It also records that error bars were not provided for all SWE-bench evaluations.
- The introduction and abstract accurately state the paper’s core contributions and key results.
- The authors discuss limitations in a dedicated section of the paper.
- The paper studies empirical LLM reasoning improvements rather than theoretical results, so proof-assumption criteria are marked not applicable.
- Data curation, technique, experimental configurations, reward implementation, and evaluation code are documented for reproducibility.
- Error bars are not included for all SWE-bench evaluations because each real-world instance is expensive to run and submissions use one attempt per example.
A Raw pull request data curation
SWE-RL constructs self-contained pull-request instances from GitHub events and repository histories, then removes contamination and noisy or unsuitable examples before RL training.
- Raw GitHub clones and events become self-contained pull-request instances through decontamination, aggregation, relevant-file prediction, and filtering.
- The curation process excludes repositories used by SWE-bench to prevent data contamination.
- GitHub events and full repository histories provide the conversational, code, and commit information needed to reconstruct pull requests.
- PR aggregation combines issues, discussions, reviews, initial code, subsequent commits, and code changes in chronological context.
- Relevant-file prediction reduces the bias of editing every presented file and improves handling of noisy context.
- Filtering removes bot-generated pull requests, empty changes, and extremely large changes to limit harmful noise.
B Agentless Mini
Agentless Mini is a simplified, scalable scaffold that decomposes issue solving into localization, repair, test generation, regression testing, and patch reranking.
- Agentless Mini emphasizes straightforward component decomposition, parallelization, and scalability.
- Localization and repair: Localization predicts relevant file paths from the issue and repository structure, generating multiple samples and consolidating unique repair sets.
- Localization and repair: During repair, the LLM generates search/replace edits from full file contents across multiple location sets.
- Reproduction tests generation and selection: Reproduction tests are sampled, filtered against the original codebase, and selected to distinguish reproduced from resolved issues.
- Regression tests selection: Regression tests are selected from initially passing tests, optionally removing tests expected to fail after the fix.
- Reranking: Patches are reranked using regression failures and reproduction-test consensus, with groups scored by |P| × |T|^2.
C Synthesizing supervised-finetuning data
The paper constructs synthetic localization and code-editing data from high-quality pull-request seeds, then filters incorrect samples using ground-truth edits and patches. These synthetic data are combined with coding and general datasets to train a supervised-finetuning baseline.
- Filtering: Synthetic samples are filtered by checking ground-truth edited files, patch correctness, search/replace formatting, and context matching.
- Seed PR collection: High-quality pull requests are selected as seeds using heuristics such as linked bug-fix issues and programming-file changes.
- Localization and editing synthesis: Localization samples prompt Llama-3.3-70B-Instruct to identify relevant files from the issue, repository structure, and file-path hints.
- Localization and editing synthesis: Code-editing samples use ground-truth pull requests and patches to synthesize search/replace edits for each issue.
- SFT baseline: Llama3-SWE-SFT-70B trains on 2B tokens with a 16k context window using synthetic localization and editing data plus coding and general SFT datasets.
D Complete prompt
The complete prompt asks the model to localize a bug from an issue and code context, reason through the task, and produce correctly formatted search/replace edits. It specifies the required response structure and editing syntax.
- Response structure: The response uses <think> for the reasoning process and <solution> for the final solution.
- Input context: The prompt presents an issue statement together with code segments from relevant files, which may contain bugs.
- Task instructions: The model must first localize the bug and then generate search/replace edits to fix the issue.
- Edit format: Each search/replace edit specifies a file path, contiguous search block, replacement block, and explicit delimiters.
- Edit format: The prompt requires each edit to be wrapped in a code block, with separate code blocks for multiple edits and proper indentation preserved.