Source-linked AI summary
SWE-smith: Scaling Data for Software Engineering Agents
John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, Diyi Yang
TL;DR
Software-engineering agents need larger, execution-validated training datasets to support open-source progress. SWE-smith builds repository environments first and synthesizes validated bug-fixing tasks at scale, yielding 50k instances across 128 repositories and a 40.2% SWE-bench Verified result for SWE-agent-LM-32B. The paper also examines training-data scale, task difficulty, repository specialization, and agent failures while releasing the toolkit and assets openly.
Problem
Open-source software-engineering agents are bottlenecked by the lack of large-scale, high-quality training data and infrastructure for collecting it.
Method
SWE-smith defines execution environments for Python codebases, synthesizes bugs that break existing tests, and creates task instances and issue descriptions automatically.
Results
40.2% on SWE-bench Verified is achieved by SWE-agent-LM-32B, setting a state-of-the-art result for open-weight models.
Takeaways & Limitations
SWE-smith provides open-source data and infrastructure for scaling software-engineering agent training and investigating agent-development phenomena.
Takeaways & Limitations
The collection pipeline is Python-centric, and the paper demonstrates only fine-tuning rather than other training techniques such as reinforcement learning.
Abstract
from arXiv · showhide
Despite recent progress in Language Models (LMs) for software engineering, collecting training data remains a significant pain point. Existing datasets are small, with at most 1,000s of training instances from 11 or fewer GitHub repositories. The procedures to curate such datasets are often complex, necessitating hundreds of hours of human labor; companion execution environments also take up several terabytes of storage, severely limiting their scalability and usability. To address this pain point, we introduce SWE-smith, a novel pipeline for generating software engineering training data at scale. Given any Python codebase, SWE-smith constructs a corresponding execution environment, then automatically synthesizes 100s to 1,000s of task instances that break existing test(s) in the codebase. Using SWE-smith, we create a dataset of 50k instances sourced from 128 GitHub repositories, an order of magnitude larger than all previous works. We train SWE-agent-LM-32B, achieving 40.2% Pass@1 resolve rate on the SWE-bench Verified benchmark, state of the art among open source models. We open source SWE-smith (collection procedure, task instances, trajectories, models) to lower the barrier of entry for research in LM systems for automated software engineering. All assets available at https://swesmith.com.
1 Introduction
SWE-smith addresses the shortage of scalable, execution-validated software-engineering training data by synthesizing task instances within repository environments. It produces a 50k-instance dataset across 128 repositories and supports SWE-agent-LM-32B’s 40.2% result on SWE-bench Verified.
- Motivation: Open-source software-engineering agents remain constrained by a lack of large-scale, high-quality training data.Infrastructure for collecting such data at scale is therefore needed to support open research.
- Motivation: GitHub pull requests and issues lack reliable validation without execution environments or tests.This limits learning to code surface form or superficial string-similarity rewards.
- Approach: SWE-smith automatically synthesizes bugs in existing repositories and validates candidates by identifying regressions that break tests.Its strategies include LM-based rewrites, AST modifications, PR inversion, and bug combination.
- Scale: SWE-smith creates 50k task instances across 128 real-world repositories while reducing the human labor and storage required for execution environments.The workflow sets up an environment, synthesizes task instances, and generates issue descriptions automatically.
- Results: 40.2% (+33.4%) on SWE-bench Verified is achieved by SWE-agent-LM-32B in a single attempt without inference-time scaling.The model is fine-tuned on 5,016 expert trajectories and sets a new state of the art for open-weight models.
- Implications: SWE-smith supports repository-specialized models that perform well on a target repository with only minor generalization loss.The paper also releases instances, environments, and trajectories as an open-source toolkit.
2 SWE-smith: Software Task Generation at Scale
SWE-smith defines execution environments before synthesizing task instances, enabling scalable bug generation and validation across Python repositories. Its dataset reaches 50k instances across 128 repositories, with realistic difficulty and substantially lower environment-storage overhead than SWE-bench-style collection.
- Collection principle: SWE-smith defines an execution environment first, then synthesizes task instances within that environment.This reverses SWE-bench’s strategy of identifying instances first and building an environment for each.
- Environment construction: Each repository is configured with passing tests, manually verified, and packaged into a Docker image.SWE-agent installs the codebase and runs its test suite; repositories are retained when more than 80% of existing tests pass.
- Task synthesis: Four candidate-generation strategies produce .diff files: LM generation, procedural AST modification, bug combination, and PR inversion.Patch combination creates multi-location tasks, while PR Mirror rewrites affected files to revert pull-request changes.
- Validation: Candidate patches are retained only when they break one or more existing passing tests, with runs exceeding two minutes discarded.The resulting failures are designated Fail-to-Pass tests for task validation.
- Scale and cost: 50k instances are generated across 128 Python repositories, averaging 381 instances per repository and reaching 2277 for pandas-dev/pandas.The reported creation cost is $1360, including $1000 for bug generation, $160 for installation, and $200 for issue generation.
- Scalability: Shared repository environments make creating 50k SWE-bench-style instances an estimated 500x more storage-efficient than per-instance environments.A comparable SWE-bench collection is estimated to require 50 to 150 TB of environment storage.
3 Experiments
The experiments use rejection-sampling fine-tuning with SWE-agent trajectories and evaluate models on SWE-bench Lite, Verified, and Multilingual. Evaluation reports the percentage of instances successfully resolved under fixed agent-system settings.
- Training setup: Rejection-sampling fine-tuning improves a base language model using trajectories generated on curated SWE-smith task instances.The workflow records expert-agent trajectories and fine-tunes a student model on them.
- Models: Claude 3.7 Sonnet serves as the primary expert, while Qwen-2.5-Coder-Instruct 7B and 32B models serve as base models.Claude 3.5 Sonnet and GPT-4o are also used for comparisons with prior work.
- Agent system: SWE-agent gives the language model an Agent Computer Interface for generating ReAct-style thought-action pairs that edit files or execute shell commands.Expert trajectories use at most 75 steps and a $2.00 cost limit; student inference uses the same step limit and temperature 0.0.
- Evaluation: Evaluation covers SWE-bench Lite’s 300 instances, Verified’s 500 instances, and Multilingual’s 300 instances across nine additional programming languages.The reported % resolved metric is the proportion of successfully resolved instances.
4 Results
SWE-smith experiments show that scaling data across trajectories and repositories improves agent performance, while task-generation choices affect training quality. Specialization boosts target-repository performance, but difficulty alone does not predict training effectiveness, and repetitive actions remain a key failure mode.
- Results: 28.2% resolve rate on SWE-bench Verified exceeds prior results by +8.2% relative to Pan et al. and +0.7% relative to Jain et al. at 500 training trajectories.This comparison uses the same training-set size as the prior works.
- Ablations of SWE-smith: 58.6%, 41.0%, and 17.0% expert resolve rates for easy, medium, and hard tasks show that difficulty correlates with solvability but not training effectiveness.Student pass@1 scores for difficulty datasets of 2/4/6/8 were 12.4%, 10.8%, 13.6%, and 12.2%.
- Ablations of SWE-smith: Increasing the number of training repositories improves general performance with an approximately logarithmic relation at 700 expert trajectories.The comparison samples Procedural Modification tasks from pools of 4, 25, 50, and 100 repositories.
- Ablations of SWE-smith: 42.4% vs. 33.3% shows that specialist-stage fine-tuning substantially improves target-repository performance, with only slight general-performance drops.The same pattern appears in single-repository fine-tuning, at 21.2% vs. 13.6%.
- Analysis: More than 25% of SWE-agent-LM-32B trajectories contain repetitive sequences of at least length 10, compared with less than 4% for Claude 3.7 Sonnet.Localization is also reported as the dominant failure mode, with runtime limits interrupting agents that stall during the workflow.
5 Related Work
Prior software-engineering LM training work spans code-completion, preference-learning, retrieval, workflow, and agent datasets, while SWE-bench emphasizes execution-based evaluation. Agent-focused workflows place more responsibility on LM planning because they impose no fixed workflow.
- SWE-bench became a de facto evaluation setting for diverse, complex, real-world software-engineering tasks.
- Agent-system research avoids imposing a fixed workflow, increasing reliance on language models to plan and refine actions.This places more emphasis on LM capabilities than on inference scaffolds.
- Training-data research also covers instruction following, preference learning, retrieval-augmented generation, workflows, and agent settings.
6 Discussion
SWE-smith combines scalable task collection with execution environments and supports the SWE-bench and SWE-agent ecosystem. Its main boundaries are Python-centric collection and limited exploration of training methods beyond fine-tuning.
- The collection pipeline is Python-centric because its program analysis and transformations rely heavily on Python’s ast library.The authors state that the collection strategy is transferable to other languages.
- The paper demonstrates SWE-smith’s effectiveness through fine-tuning but does not explore reinforcement learning or other training techniques.This limitation reflects compute and budget constraints and the paper’s primary focus as a dataset contribution.
- The appendix describes infrastructure, collection strategies, trajectory creation, ablations, and additional results.
- SWE-smith provides execution environments and task instances that integrate with SWE-bench evaluation and SWE-agent trajectory generation.
A.1 SWE-smith Task Instance
A SWE-smith task instance packages a repository, bug-inducing patch, generated problem statement, validation tests, and metadata. The pipeline installs repositories, synthesizes bugs through multiple strategies, and validates usable instances.
- A SWE-smith task instance includes repository, instance ID, base commit, bug patch, problem statement, timestamp, and FAIL TO PASS tests.
- The bug patch is applied to the original codebase, and reverting it constitutes the solution.
- FAIL TO PASS tests are the unit tests broken by the applied bug patch, while PASS TO PASS tests remain unbroken.
- Unlike SWE-bench, SWE-smith omits hints and hidden test patches, and assigns created-at timestamps when bugs are successfully validated.Installation instructions are specified for each repository and commit.
- Repositories are predominantly Python projects, and the pipeline uses assumptions about packaging and testing to make automated setup more tractable.Repositories must also permit non-proprietary use.
- Bug-generation strategies include LM rewrites, AST-based procedural modifications, and combinations of bugs.Procedural modifications use controlled likelihood to regulate how often transformations are applied.
B.4 Pull Request Mirroring
SWE-smith’s PR-mirroring strategy uses language models to reverse pull-request changes in current repository states, creating validated bug instances without reconstructing historical environments. It recovers most tested Django instances but can leave repository context out of sync.
- PR mirroring reverses pull-request changes by prompting a language model to rewrite each affected file in the current repository state.Reasoning models are reported as particularly effective because direct git patch reversal often fails when code locations have changed.
- 92 of 100 sampled Django task instances were recovered, including 84 with identical FAIL TO PASS tests.The remaining 8 recovered instances broke a subset because some tests had been removed over time.
- PR mirroring removes instance-specific Docker images and past-version installation specifications by mirroring pull requests against one repository version.
- The strategy relaxes SWE-bench-style filtering requirements by tolerating missing issue text or testing changes when validation can still establish a broken test.
- PR mirroring is constrained because the rest of the repository can be out of sync with the codebase state when the pull request was applied.This can affect issue reproducibility and issue-description accuracy; an earlier repository commit is offered as a mitigation.
C.1 Bug Generation Statistics
SWE-smith’s bug-generation strategies achieve higher collective yield than SWE-bench while producing diverse, adjustable task distributions and using substantially fewer Docker images.
- Yield rates: 13.18% is the lowest reported yield rate, achieved by PR Mirroring, versus SWE-bench’s 2.46%.LM-based intentional function modification yields more bugs than best-effort rewriting, while procedural efficacy varies by strategy.
- Yield rates: SWE-smith’s collective yield rate is significantly higher than SWE-bench’s collection strategy.
- Yield rates: Lower test coverage generally correlates with lower yield rates across repositories.Individual strategies also have repository-specific preconditions, such as requiring Python classes or multi-operation expressions.
- Dataset characterizations: SWE-smith instances often break multiple Fail-to-Pass tests and collectively affect a larger share of a repository’s test suite than prior datasets.Edited lines and files follow trends highly similar to SWE-bench Verified.
- Dataset characterizations: SWE-smith’s task distributions are adjustable because its flexible bug-generation techniques can reshape the distributions of tests, lines, and files edited.For example, generating more combined patches shifts all three Figure 17 curves.
- Execution environments: 290.54 GB across 125 Docker images supports more than 20x SWE-smith’s bugs, compared with 1.2 TB for SWE-bench’s 2,294 instances.Images are shared by bugs generated from the same repository and commit, reducing storage per task instance.
C.2 Case Study: SWE-bench & SWE-smith
A pallets/flask case study finds that SWE-smith produces many more validated bugs with far less human effort and cost than SWE-bench-style collection, while remaining unsuitable for evaluation as presented.
- SWE-bench collection: 11 SWE-bench task instances represent 0.45% of 2,434 pallets/flask pull requests after filtering.
- Collection effort: SWE-smith reduces pallets/flask collection from an estimated 38 hours to repository-agnostic automated generation and validation.Function-level rewriting took 23 minutes and cost $2.47, while validation filtered 135 unqualified candidates in 14 minutes.
- Collection effort: 207.27 minutes per instance for SWE-bench-style collection compares with 0.176 minutes and approximately $0.00613 using SWE-smith.
- Bug coverage: 422 of 474 tests are broken by SWE-smith instances, yielding 89.03% bug coverage versus 15 of 207 tests and 7.25% for SWE-bench.The comparison uses 267 SWE-smith instances and 11 SWE-bench instances from pallets/flask.
- Evaluation boundary: SWE-smith is not appropriate for evaluation as presented because Fail-to-Pass tests remain available at inference time and issue text is not checked for leakage or underspecification.The authors suggest deleting those tests and validating issue ambiguity and leakage as possible amendments.
- Issue generation: SWE-smith’s four issue-generation strategies include LM-generated issues, fixed templates, Fail-to-Pass code and logs, and original issue text.LM-generated issues use the patch, tests, source code, execution logs, and a SWE-bench Verified demonstration.
E Difficulty Rating
The paper rates task difficulty with a learned easy/medium/hard classifier and finds that SWE-smith can generate instances across a range of difficulties depending on bug-generation strategy.
- Difficulty model: The model predicts three labels: easy for under 15 minutes, medium for 15 minutes to 1 hour, and hard for more than 1 hour.The paper maps these labels to difficulty scores of 1, 5, and 9 and averages them across task instances.
- Difficulty model: Difficulty scores are computed as the average numeric score across all task instances.The score mapping is easy/medium/hard = 1/5/9.
- Findings: LM Modify tasks are consistently rated easy, while Procedural Modifications are next easiest and PR Mirrors and LM Rewrites are harder.The paper attributes the easy LM Modify results mainly to simple variable-assignment mistakes and finite procedural bug types.
- Findings: Aggregating smaller functions produces bugs rated as more complex, consistent with larger numbers of edited functions and files being harder.
- Findings: SWE-smith can create task instances spanning a range of difficulties.
F.2 Evaluation Datasets
The paper introduces SWE-bench Multilingual as a compact, SWE-bench-compatible benchmark across nine languages, motivated by the limits of Python-focused evaluation; existing state-of-the-art performance is markedly worse on it.
- Related benchmarks: SWE-bench contains 2,294 task instances from 12 predominantly Python GitHub repositories, while SWE-bench Multimodal covers 12 predominantly JavaScript and TypeScript repositories.SWE-bench Multimodal’s evaluation dataset consists of 510 task instances.
- Dataset composition: 300 task instances across 42 repositories and 9 programming languages comprise SWE-bench Multilingual.The languages include JavaScript, TypeScript, C, C++, Go, Java, PHP, Ruby, and Rust.
- Motivation: SWE-bench Multilingual evaluates agents across varied programming languages and application domains without visual inputs.The benchmark addresses Python-specific tooling and the visual-input confound in SWE-bench Multimodal.
- Design goals: SWE-bench Multilingual remains fully compatible with SWE-bench so existing users can adopt it without changing infrastructure.
- Design goals: The dataset is intentionally kept small enough to run quickly.The authors explicitly constrain its size despite concurrent multilingual datasets with more instances.
- Results: Existing state-of-the-art methods perform markedly worse on SWE-bench Multilingual.The paper presents this result as motivation for extending SWE-smith toward better multilingual agentic coding models.
F.4 Training Analyses
The training analyses examine inference scaling, trajectory selection, multilingual transfer, runtime limits, failure modes, and scaffold interventions for SWE-agent-LM-32B. Results show improved performance with more runs and repositories, but multilingual performance and repetitive behavior remain important limitations.
- Pass@k trend: Higher k increases the percentage of resolved SWE-bench Verified instances for SWE-agent-LM-32B.The reported Pass@1 score was calculated from six runs, while Figure 22 examines performance at higher k values.
- Rejection sampling fine-tuning: Rejection sampling fine-tuning outperforms random trajectory sampling on the downstream SWE-bench Verified task.The ablation compares filtered training trajectories against randomly sampled points using the same student model.
- Multilingual performance: SWE-agent-LM-32B performs poorly on multilingual coding, achieving 8.4% Pass@1 versus 43% for Claude 3.7 Sonnet.The multilingual evaluation covers 300 task instances; Qwen 2.5 Coder Instruct achieves 6.5% Pass@1.
- Turn counts and cost: Runtime and step limits strongly affect average costs, turn counts, and the number of successful submissions.Most failed instances terminate because of cost or turn limits, and resolved-instance counts vary with the prescribed step limit.
- Failure modes: Failure analysis categorizes errors, early termination, localization, editing, and successful submissions according to the agent workflow and termination conditions.The procedure distinguishes runtime or context errors from being stuck during localization, reproduction, or editing, then evaluates submitted patches against the gold patch.
- Mitigating repetitive actions: Scaffold interventions reduce repetitive actions but slightly lower resolved instances to 192 (38.4%), suggesting repetition may be a symptom of difficult instances.The interventions add warnings, resample actions, and alter assistant-message generation or temperature after repeated commands.
G Miscellaneous
The miscellaneous material describes how the SWE-smith and SWE-bench scaling comparison was constructed, reviews related trajectory-generation work, and documents practical dataset-construction costs. It also notes that the methodology requires only limited human labor.
- Teaser figure construction: The Figure 1 scaling comparison counts SWE-smith instances per repository and applies SWE-bench’s candidate-collection script across the same 128 repositories.The repositories are sampled at intervals of five and sorted by number of stars.
- Trajectory analysis: The analysis measures repetitive trajectories using the fraction containing repetitive action sequences longer than L.Base commands are normalized by removing environment-variable manipulation, retaining the final chained command, and removing arguments.
- Extended related works: Prior web-navigation work also generates training trajectories through strategies including random walks and related trajectory-generation techniques.The discussion situates SWE-smith among methods for improving interactive capabilities of open-source models.
- Human labor: SWE-smith requires around 8 minutes of human labor in addition to automated execution-environment construction and unit-test generation influences.The passage contrasts this requirement with the substantially greater labor associated with SWE-bench-style collection.