PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-smith: Scaling Data for Software Engineering Agents

John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, Diyi Yang

arXiv:2504.21798v2cs.SEcs.AIcs.CL

TL;DR

Open-source software-engineering agents lack large-scale, high-quality training data. SWE-smith builds execution environments before synthesizing software tasks, and its resulting SWE-agent-LM-32B reaches 40.2% (+33.4%) on SWE-bench Verified in a single attempt, a new open-weight state of the art.

  • Problem

    Open-source software-engineering agents lack large-scale, high-quality training data, while GitHub-derived tasks often lack execution environments or tests for reliable solution validation.

    Source: §1 · pp. 1–2

  • Method

    SWE-smith builds execution environments first, then synthesizes test-breaking bugs in GitHub codebases through LM rewrites, AST edits, PR reversals, and bug combinations.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    40.2% (+33.4%) on SWE-bench Verified in a single attempt, setting a new state of the art for open-weight models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The dataset’s scale and diversity support investigation of SWE-agent development trends, including how training on more instances, bug types, and repositories helps.

    Source: §1 · p. 2

    SWE-smith’s collection pipeline is Python-centric, although its collection strategy is transferable to other languages.

    Source: §6 · p. 10

Abstract

from arXiv · show

Despite recent progress in Language Models (LMs) for software engineering, collecting training data remains a significant pain point. Existing datasets are small, with at most 1,000s of training instances from 11 or fewer GitHub repositories. The procedures to curate such datasets are often complex, necessitating hundreds of hours of human labor; companion execution environments also take up several terabytes of storage, severely limiting their scalability and usability. To address this pain point, we introduce SWE-smith, a novel pipeline for generating software engineering training data at scale. Given any Python codebase, SWE-smith constructs a corresponding execution environment, then automatically synthesizes 100s to 1,000s of task instances that break existing test(s) in the codebase. Using SWE-smith, we create a dataset of 50k instances sourced from 128 GitHub repositories, an order of magnitude larger than all previous works. We train SWE-agent-LM-32B, achieving 40.2% Pass@1 resolve rate on the SWE-bench Verified benchmark, state of the art among open source models. We open source SWE-smith (collection procedure, task instances, trajectories, models) to lower the barrier of entry for research in LM systems for automated software engineering. All assets available at https://swesmith.com.

1 Introduction

SWE-smith automatically creates validated software-engineering tasks at scale by synthesizing bugs in real Python codebases. Its 50k-instance dataset supports SWE-agent-LM-32B, which achieves 40.2% on SWE-bench Verified.

1 Introduction

  • 40.2% (+33.4%) on SWE-bench Verified in a single attempt without inference-time scaling makes SWE-agent-LM-32B a new open-weight state of the art.The model was fine-tuned on Qwen 2.5 Coder Instruct 32B using 5,016 expert trajectories generated with Claude 3.7 Sonnet.Source: Figure 1 · p. 1, §1 · p. 2
  • Unlike PR- and issue-crawled data without tests, and SWE-bench-style collection constrained by scarce qualifying PRs and manual environment setup, SWE-smith combines flexible environments with scalable collection.Without execution environments or tests, crawled instances cannot reliably validate solutions; SWE-bench-style collection can support trajectory filtering by unit tests but requires substantial human intervention to set up environments.Source: §1 · p. 2
  • SWE-smith creates 50k task instances across 128 real-world GitHub repositories by setting up environments, synthesizing bugs, and generating issue descriptions with LMs.It synthesizes hundreds to thousands of tasks per codebase using errant LM rewrites, procedural AST modifications, undone PRs, and combinations of bugs; execution tests identify candidates that cause regressions.Source: §1 · p. 2, Figure 2 · p. 2
  • SWE-smith’s scale and diversity support findings that more training instances, bug types, and repositories help, generated issue text approximates real issues, and repository-specific optimization incurs only minor generalization loss.The introduction presents these as phenomena made investigable by the dataset.Source: §1 · p. 2
  • SWE-smith is released as an open-source toolkit with task instances, execution environments, and trajectories to support stronger open-source LM agents.The paper identifies the lack of large-scale, high-quality training data as a bottleneck for open-source software-engineering LMs.Source: §1 · pp. 1–2

2 SWE-smith: Software Task Generation at Scale

SWE-smith defines a repository’s execution environment first, then synthesizes and validates task instances within it. It scales this approach across repositories while sharing environments among tasks from the same repository.

2.1 Collection

  • Repositories are installed and tested at their latest commit, manually verified, required to have more than 80% of existing tests pass, and packaged as Docker images.The target pool starts from the 5,000 most-downloaded PyPI packages and excludes packages with fewer than 1,000 GitHub stars and the 12 SWE-bench test repositories.Source: §2.1 · p. 3
  • Candidate generation uses LM Modify and LM Rewrite, AST-based procedural modifications, patch combination across files or modules, and PR Mirror to revert pull-request changes.PR Mirror prompts an LM to rewrite affected files based on PR diffs rather than checking out the PR’s base commit.Source: §2.1 · p. 3
  • Candidates are retained only if applying a patch breaks at least one previously passing test, and candidates whose test runs exceed two minutes are discarded.These failures are called Fail-to-Pass (F2P) tests.Source: §2.1 · p. 3
  • Issue-style task statements are generated from each patch, a random F2P test, and test output with the bug applied, with reproduction code based on that test.Source: §2.1 · p. 3
  • Manual parsing of installation instructions takes about 7 minutes per repository, and parsing test outputs about 1 minute; one author created SWE-smith with about 20 hours of labor.Test-output parsers can be reused across repositories with the same testing infrastructure, and SWE-smith avoids determining installation specifications for multiple codebase versions.Source: §2.1 · pp. 3–4

2.2 Features

  • LM-based strategies differ in cost and yield: PR Mirrors cost more because they rewrite entire files, while LM Modify yields more than LM Rewrite, which is not explicitly prompted to introduce bugs.Yield is limited by insufficient test coverage or candidates that do not introduce relevant issues.Source: §2.2 · p. 4
  • 50k instances across 128 repositories used shared repository environments; for 50k tasks, SWE-bench-style environments were estimated to require 50–150 TB, a 500x difference.SWE-smith generated an average of 381 instances per repository, with as many as 2,277 for pandas-dev/pandas.Source: Table 2 · p. 4, §2.2 · pp. 4–5
  • Average difficulty scores were 5.27–5.72 for SWE-smith strategies, compared with 5.01 for SWE-bench and 5.62 for SWE-gym.A Qwen 2.5 32B model rated difficulty after training on human-annotated examples, achieving 75.3% test accuracy.Source: §2.2 · pp. 4–5

3 Experiments

The experiments use rejection-sampling fine-tuning on SWE-smith tasks to train student models, then evaluate the resulting agent on separate test splits across SWE-bench variants.

Experiment workflow

  • The workflow curates SWE-smith tasks, records expert-agent trajectories, fine-tunes the student on resolved instances, and evaluates it on a separate test split.This uses rejection sampling fine-tuning (Yuan et al., 2023).Source: §3 · p. 5

Models

  • Expert models are Claude 3.7 Sonnet, Claude 3.5 Sonnet, and GPT-4o; Qwen-2.5-Coder-Instruct 7B and 32B are the student base models.Claude 3.5 Sonnet and GPT-4o are included for comparisons with prior works (Pan et al., 2024).Source: §3 · p. 5

Agent system

  • SWE-agent (Yang et al., 2024a) supplies an Agent Computer Interface and prompts ReAct (Yao et al., 2023b) thought-action pairs for code edits or shell commands.Expert trajectory runs are capped at 75 steps and $2.00; student inference uses the same step cap and temperature 0.0.Source: §3 · p. 5

Evaluation metrics

  • Evaluation reports the % resolved metric on SWE-bench Lite (300 instances), Verified (500), and Multilingual (300 instances across nine additional languages).Lite is curated as an easier, less costly split; Verified is human-curated for clearer problem statements and more reliable evaluation.Source: §3 · p. 5

4 Results

SWE-agent-LM-32B, fine-tuned on SWE-smith trajectories, achieves state-of-the-art performance in the reported comparison, and performance improves with more training data. Ablations identify effective data-generation choices, while agent analysis highlights efficiency alongside recurring failure modes.

4 Results

  • SWE-agent-LM-32B achieves state-of-the-art performance among the compared systems on SWE-bench Lite and Verified.Table 3 reports pass@1 resolve rates for existing solutions and models fine-tuned on SWE-smith; it excludes systems using verifiers or multiple test-time attempts.Source: §4 · p. 5, Table 3 · p. 6
  • 6,457 expert trajectories were collected from 8,686 task instances, a 36% resolve rate across 17,906 attempts, before filtering to 5,016 training points.The final set limits each instance to three trajectories; performance increases with more trajectories.Source: §4 · p. 6
  • At 500 successful trajectories, the model reaches 28.2% resolve rate on SWE-bench Verified, with relative differences of +8.2% versus Pan et al. (2024) and +0.7% versus Jain et al. (2025).The comparison uses a training set size reported by both prior works.Source: §4 · p. 6

4.1 Ablations of SWE-smith

  • PR mirrors were most effective, Procedural Modification and LM Rewrite were competitive, and LM Modify dropped sharply; LM-generated issues matched original text, while test-based issues reduced reproduction.On 500 Verified instances, the LM-issue-trained student attempted bug reproduction in 379 runs, versus 127 for the test-based-issue model, a 66% decrease.Source: §4.1 · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • Expert resolve rates fell from 58.6% (easy) to 41.0% (medium) to 17.0% (hard), but student performance showed no strong correlation with training difficulty.For difficulty scores 2, 4, 6, and 8, student pass@1 scores were 12.4%, 10.8%, 13.6%, and 12.2%, respectively, on SWE-bench Verified.Source: §4.1 · p. 7
  • More training repositories improve performance approximately logarithmically, while SymPy specialization boosts target-repository results with only slight generalization losses.Target-repository performance was 21.2% versus 13.6% for single-repository fine-tuning and 42.4% versus 33.3% for specialist-stage fine-tuning.Source: Figure 5 · p. 7, Figure 4 · p. 7, §4.1 · p. 8

4.2 Analysis of Agent Behavior

  • SWE-agent-LM-32B resolves tasks in fewer average steps than Claude 3.7 Sonnet (24.9 vs. 29.1), and uses 24.8 vs. 25.6 steps on tasks both resolve.Its naturally terminating submissions nearly match Claude 3.7 Sonnet in resolution-detection accuracy (60% vs. 63%); 31 instances were resolved after at least 40 steps.Source: §4.2 · pp. 8–9, Figure 6 · p. 8
  • More than 25% of SWE-agent-LM-32B trajectories contain repetitive sequences of at least 10 actions, versus less than 4% for Claude 3.7 Sonnet; length-10 sequences correspond to 89% failure probability.More than half of unresolved instances are terminated by cost or step limits, frequently before source code has been modified.Source: §4.2 · p. 9, Figure 8 · p. 9

5 Related Work

Related work motivates SWE-smith’s focus on agent trajectories and repository-level task generation: SWE-bench has become a prominent evaluation setting, while existing training-data approaches vary in task type and scale. The paper contrasts its method with workflow, RePOST, and R2E-Gym approaches.

LMs for Software Engineering

  • SWE-bench’s real-world challenges have made it a de facto evaluation setting, but workflows may generalize poorly beyond Python, motivating focus on LM-agent trajectories.Workflows decompose tasks into human-engineered sub-goals; without imposed workflows, agents rely more on LM planning and action refinement (Sumers et al., 2024; Yang et al., 2023; Yao et al., 2023b).Source: §5 · pp. 9–10

Training Datasets for Coding

  • Prior coding datasets cover completion, retrieval, workflows, and SWE-bench agents, while SWE-smith adapts repository-level code breaking to reduce task-definition and environment-building effort.RePOST (Xie et al., 2025b) removes the original codebase context when sandboxing functions and evaluates code generation; R2E-Gym (Jain et al., 2025) uses SWE-bench’s pipeline with augmentations, and its 51% resolve rate is not comparable because each instance is attempted 26 times. The authors state they are first to address the limited scalability of previous approaches.Source: §5 · p. 10

6 Discussion · Appendix · A Infrastructure

The discussion reports SWE-smith’s scale and SWE-agent-LM-32B result while delimiting its Python-centric, finetuning-only scope; the appendix maps the pipeline and its supporting infrastructure and dataset analyses.

6 Discussion

  • The pipeline is Python-centric but transferable; compute and budget constraints and dataset focus limited training demonstrations to finetuning, leaving reinforcement-learning approaches unexplored.Source: §6 · p. 10
  • 40.2% on SWE-bench Verified is the state-of-the-art result reported for SWE-agent-LM-32B trained with SWE-smith.Source: §6 · p. 10
  • SWE-smith provides 50k task instances from 128 real-world GitHub repositories, scaling instances, environments, and trajectories at a fraction of prior costs without sacrificing faithfulness to open-source development practices.Source: §6 · p. 10

Appendix

  • The appendix covers infrastructure and collection strategies in Sections A–D, then trajectory-dataset creation and additional ablations and results in later sections.It compares SWE-smith with existing datasets, including SWE-bench and SWE-gym.Source: Appendix · p. 16
  • Figure 9 shows SWE-smith constructing environments and task instances, SWE-agent generating trajectories and running inference, and SWE-bench evaluating models.Source: Figure 9 · p. 16

A Infrastructure

  • Infrastructure coverage includes task-instance form, repository identification and automatic-install configuration, and the task-validation and evaluation harnesses.Source: A Infrastructure · p. 16, Appendix · p. 16

A.1 SWE-smith Task Instance … Combine Bug Patches

SWE-smith defines bug tasks around validated patches and failing tests, then generates and combines bugs through language-model and procedural strategies. Its workflow covers repository setup, task conversion, and combinations of bugs within files or modules.

A.1 SWE-smith Task Instance

  • A SWE-smith task records a bug-causing patch, problem statement, and FAIL TO PASS and PASS TO PASS tests; unlike SWE-bench, it has no hidden test patch.All FAIL TO PASS bugs are visible and runnable at inference time.Source: §A.1 · p. 17

A.2 Repository Selection

  • SWE-smith selects predominantly Python repositories to make automated installation and testing more tractable, and checks that licenses permit the work’s non-proprietary use.Repository setup averages $0.72 and 17 agent steps, followed by 3–20 minutes of manual review; one author spent an estimated 18 hours on 128 repositories.Source: §A.2 · pp. 17, 19

A.3 Validation, Evaluation Harnesses

  • SWE-smith adapts SWE-bench’s validation script to apply each bug patch on a mirror branch and populate task fields, including Fail-to-Pass and Pass-to-Pass tests, from validation logs.Source: §A.3 · p. 19

B Bug Generation Strategies

  • Bug-generation strategies are selected for codebase-agnostic operation, reliable usable tasks with at least one failing test, and controllable quantity and quality.Source: B Bug Generation Strategies · p. 20

B.1 Generating with an LM

  • For LM generation, SWE-smith prompts o3 mini to rewrite or reimplement functions, or adapt the approach to classes, while discouraging inline bug comments and trivial syntax or compilation errors.Reimplementation removes the original function body and asks the model to preserve its signature.Source: §B.1 · pp. 21–22

B.2 Procedural Modification

  • Procedural generation mutates function or class ASTs after filtering candidates by modification-specific criteria, with a likelihood parameter controlling how often each change is applied.Complexity filters use counts of conditionals, loops, boolean operators, exception-handling blocks, and comparisons to exclude simple functions or, sometimes, long monolithic ones.Source: §B.2 · pp. 22–23

B.3 Combine Bug Patches

  • SWE-smith combines validated LM- or procedurally generated bugs from the same file, applying patches sequentially and saving successful combinations as a single revalidated bug.It removes remaining combinations that reuse a patch, limits combinations per file and search size, and typically uses 2–4 bugs, at most 3 saved combinations per file, and up to 40 combinations.Source: §B.3 · pp. 24–25

Combine Bug Patches

  • For multi-file combinations, SWE-smith groups bugs by source submodule and saves only combinations changing at least two files; it uses depth 2 because deeper groupings sharply reduce yield.The authors note that 75% of SWE-bench instances changing at least two files modify files within the same submodule.Source: §B.3 · pp. 25–26

B.4 Pull Request Mirroring · C Dataset Statistics

PR mirroring uses language models to reverse real pull-request changes and recovered most sampled Django tasks; the dataset statistics describe repositories spanning seven software categories.

B.4 Pull Request Mirroring

  • Directly applying git apply --reverse failed for most inspected patches because relevant code had moved, although most PR changes remained in the codebase.The authors inspected 100 PR patches from sqlfluff/sqlfluff; reasoning models such as o3-mini were particularly effective at reversal.Source: §B.4 · p. 26
  • For each changed file, an LM uses the PR-specific changes and the file’s current source to generate a rewrite that reverses the changes, which are aggregated into one patch.The workflow applies the resulting changes to the codebase and extracts the reversal patch.Source: §B.4 · p. 26, Figure 16 · p. 26
  • PRs are crawled from January 1, 2023 onward, with some repositories using older PRs; the method excludes PRs changing more than 8 files.The file limit reflects that no SWE-bench instance editing more than 6 files had been solved, according to the cited work.Source: §B.4 · pp. 26–27
  • 92 of 100 sampled Django task instances were recovered, and 84 broke identical F2P tests while 8 broke a subset because tests had been removed.The authors present this sanity check as evidence that PR mirroring works as intended.Source: §B.4 · p. 27
  • Mirroring PRs against the same repository version removes the need for instance-specific Docker images and installation specifications for past repository versions.The paper describes past-version installation specifications as typically the most laborious task-construction step in SWE-bench.Source: §B.4 · p. 27
  • Unlike SWE-bench, PR mirroring does not require associated GitHub issues or test-related file changes, since issue text can be generated and validation checks whether tests break.SWE-bench’s listed requirements also include editing at least one code file.Source: §B.4 · p. 27
  • A downside is that the rest of the repository may be out of sync with the PR’s original state, affecting reproducibility or issue-description accuracy; the authors hypothesize 10x fewer human hours and an estimated $126.17 cost to reproduce SWE-bench.They suggest using a Docker image from an earlier repository commit as a mitigation, but did not conduct a targeted experiment of the estimate.Source: §B.4 · p. 27

C Dataset Statistics

  • SWE-smith’s repositories fall into seven categories: Data Parsing and Transformation (39), Web & API Development (11), Code Quality & Testing (12), Visualization & Presentation (8), System Tools & Protocols (17), Natural Language Processing (7), and Miscellaneous (6).The authors say this represents wider and more varied coverage of software tools and applications than prior work.Source: C Dataset Statistics · p. 27

C.1 Bug Generation Statistics

SWE-smith’s bug-generation strategies produce varied yields and task-instance characteristics, while its flexible, repository-commit-based execution environments support a larger dataset with fewer Docker images. Its tasks also differ from pull-request-derived datasets in test failures and can be adjusted to shape task distributions.

Yield rates

  • PR Mirroring yielded 13.18%, versus SWE-bench’s 2.46%; conditional inversion yielded 47.04%, versus 1.93% for class-function shuffling.Across SWE-smith’s bug-generation strategies, the collective yield rate is significantly higher than SWE-bench’s collection strategy.Source: Table 10 · p. 32, §C.1 · p. 32
  • Lower test coverage generally correlates with lower repository-level yield, and some strategies apply only when their required code structures are present.Class-based procedural methods cannot produce bugs in repositories without classes, while operation-break-chain mutations require expressions with at least two operations.Source: §C.1 · pp. 32–33, Table 11 · p. 32

Dataset characterizations

  • LM-based strategies rewrite one function in one file, while procedural and patch-combination strategies can edit multiple functions or files.Combining patches from the same file always edits at least two functions; combining across modules edits at least two files.Source: §C.1 · p. 33, Table 12 · p. 33
  • SWE-smith has a higher proportion of tasks with multiple Fail-to-Pass tests and causes failures across more of a repository’s test suite, while its edited-line and file trends resemble SWE-bench Verified.The authors suggest that these broader test failures may expose models to more codebase functionality.Source: §C.1 · p. 33, Figure 17 · p. 33
  • Unlike pull-request-derived datasets, SWE-smith’s task distributions can be shaped by choosing bug-generation strategies, potentially providing models broader codebase exposure.Generating more tasks through patch combination would shift all three Figure 17 distributions; the authors contrast this flexibility with attributes constrained by real-world software development behavior.Source: §C.1 · p. 33

Continuation of scaling execution environments

  • With more than 20x the bugs, SWE-smith uses 125 Docker images totaling 290.54 GB, compared with SWE-bench’s 2,294 images requiring 1.2 TB.Bugs generated for the same repository and commit can share one image because their installation and testing procedures are shared.Source: §C.1 · p. 34

C.2 Case Study: SWE-bench & SWE-smith · D Issue Generation

The pallets/flask case study finds that SWE-smith produces more task instances with less collection effort and greater bug coverage than SWE-bench, but its instances are not evaluation-ready as presented. The issue-generation section describes four strategies that vary the information supplied in task statements.

C.2 Case Study: SWE-bench & SWE-smith

  • 267 of 402 SWE-smith candidates were valid pallets/flask task instances, versus 11 SWE-bench instances (0.45% of 2,434 PRs).SWE-smith used function-level rewriting; SWE-bench’s 11 instances came from 107 PRs meeting its filtering criteria.Source: §C.2 · p. 34
  • Per instance, SWE-bench-style collection took 207.27 minutes, versus 0.176 minutes and $0.00613 in API costs using SWE-smith.For SWE-smith, defining installation and testing specifications took 10 minutes, rewriting took 23 minutes and $2.47, and validation took 14 minutes.Source: §C.2 · pp. 34–35
  • SWE-smith broke 422 of 474 unique tests (89.03% bug coverage), compared with 15 of 207 tests (7.25%) for SWE-bench.The test counts differ partly because pallets/flask had increased test coverage by Nov. 28, 2024, compared with June 2023; the authors attribute the coverage gap to most tests accompanying working code rather than capturing existing errant behavior.Source: §C.2 · p. 35
  • As presented, SWE-smith is unsuitable for evaluation because it lacks hidden tests, exposes Fail-to-Pass tests at inference, and does not check issue ambiguity or solution leakage.The authors suggest removing Fail-to-Pass tests and validating issue text, while calling for analysis of how faithfully synthetic bugs represent real-world issues and PRs.Source: §C.2 · p. 35

D Issue Generation

  • LM-generated issues use a SWE-bench Verified example, the bug patch, Fail-to-Pass tests and their code and execution logs to describe the bug.The prompt asks for an issue in the style of the selected SWE-bench Verified problem statement.Source: D Issue Generation · p. 35
  • Seven fixed issue templates vary the supplied bug information, with the most informative template assigned the highest likelihood to encourage well-specified task statements.The templates draw on bug type, changed files and functions, and Fail-to-Pass tests; Figure 19 compares difficulty distributions for SWE-bench-style datasets and SWE-smith, with average scores shown above bars.Source: D Issue Generation · p. 35, Table 13 · p. 36, Figure 19 · p. 37
  • A separate strategy provides code and execution logs for one randomly selected Fail-to-Pass test, while PR-mirrored tasks can use associated original issue text.Other Fail-to-Pass tests are withheld in the code-and-logs strategy; 708 of 2,345 PR-mirrored task instances (30.19%) had at least one associated GitHub issue.Source: D Issue Generation · pp. 35–36

E Difficulty Rating · F Experiments · F.1 Training Details

The paper uses a learned difficulty classifier to compare the complexity of SWE-bench-style tasks, then details the training and inference configurations used in its experiments. The difficulty analysis finds that bug-generation strategies produce tasks of varying difficulty, while the experiments describe rejection-sampling fine-tuning and SWE-agent setup.

E Difficulty Rating

  • The three-label difficulty model achieves 75.3% test accuracy, and all its errors are off by one category.It is trained on 1,699 annotations, merging the 1–4-hour and 4+-hour categories into 1+ hour; the 80/20 split preserves the original distribution, and the model uses the problem statement and solution patch as input.Source: E Difficulty Rating · p. 37
  • Difficulty is scored as the average of instance scores, assigning 1, 5, and 9 to easy, medium, and hard labels.The model grades instances after their bug patch and issue text have been created.Source: E Difficulty Rating · p. 37
  • LM Modify tasks are consistently easiest, PR Mirrors and LM Rewrites are harder, and combining smaller functions yields more complex tasks; SWE-smith produces a range of difficulty levels.Manual spot checks attribute LM Modify’s easy ratings largely to simple variable-assignment mistakes; the harder ratings for PR Mirrors and LM Rewrites are also consistent with lower resolve rates by SWE-agent + Claude 3.7 Sonnet.Source: E Difficulty Rating · pp. 37–38

F Experiments

  • Experiments report configuration details and additional analyses of the SWE-smith dataset and agents trained on it.For rejection-sampling fine-tuning, the authors train only on expert trajectories corresponding to resolved instances, using full-parameter tuning with a 5e-5 learning rate, up to 3 epochs, and a 32768-token context length on 2–8 NVIDIA H100 80G GPUs.Source: F Experiments · p. 38, §F.1 · p. 38

F.1 Training Details

  • The expert and student SWE-agent configurations differ in response elicitation, parsing, message-size constraints, and system prompt, while sharing the task prompt, feedback format, and three-tool interface.The tools are bash, a string-replacement editor, and submit.Source: §F.1 · p. 38
  • Expert function-call trajectories must be converted to the student’s XML-tagged thought-and-action format before fine-tuning.Direct fine-tuning on the expert trajectories in their original format does not work.Source: §F.1 · p. 39
  • The authors use the SWE-agent and Claude 3.7 Sonnet setup based on its reported state-of-the-art open-source-agent performance, without repeating a configuration hyperparameter sweep.They note that other tool designs and formatting calls could be explored.Source: §F.1 · p. 39
  • Expert and student runs terminate at 75 steps, a $2.00 cost limit, or the model’s context limit; student inference retains five recent tool outputs and uses temperature 0.0.The authors say most automatic expert-run terminations occur at the 75-step limit; the student cost limit uses the GPT-4o cost function as of April 2025.Source: §F.1 · pp. 39–40

F.2 Evaluation Datasets · F.3 Trajectory Dataset Breakdown

The paper introduces a compact, SWE-bench-compatible multilingual evaluation set and describes the scope and composition of the SWE-agent trajectory dataset used for training.

F.2 Evaluation Datasets

  • SWE-bench contains 2,294 tasks from 12 predominantly Python repositories, but the authors omit its full test set because evaluation is costly and comparable references are limited.SWE-bench evaluates issue resolution from codebases and issue descriptions (Jimenez et al., 2024b).Source: §F.2 · p. 40
  • SWE-bench Multimodal has 510 tasks from 12 predominantly JavaScript and TypeScript repositories, but SWE-agent-LM-32B is not evaluated on it because it cannot process images.Its tasks are associated with issues containing visual assets (Yang et al., 2024b).Source: §F.2 · p. 40
  • 300 task instances across 42 repositories and 9 languages make up the new SWE-bench Multilingual evaluation dataset.A single author applied SWE-bench’s collection strategy; task instances average 48 lines edited, and the median number of Fail-to-Pass tests is one.Source: §F.2 · p. 40
  • The dataset targets cross-language and cross-domain evaluation, remains SWE-bench-compatible, and is deliberately small for quick runs.Its motivation includes reducing reliance on Python-specific tooling, while avoiding the visual-input confound in SWE-bench Multimodal.Source: §F.2 · p. 41
  • Existing state-of-the-art SWE-bench methods perform markedly worse on SWE-bench Multilingual, motivating directions for improving agentic coding models.The paper discusses this finding in §F.4 and identifies extending SWE-smith as a potential next step.Source: §F.2 · p. 41

F.3 Trajectory Dataset Breakdown

  • 5,000 trajectories were curated for SWE-agent-LM-32B training, with a cap of 3 successful trajectories per task instance to guard against easy-data bias.Most trajectories used claude-3-7-sonnet-20250219; a minority used claude-3-5-sonnet-20240620 and gpt-4o-2024-08-06.Source: §F.3 · p. 41
  • The final training dataset represents 123 repositories, including at least 10 trajectories from 91 repositories, and covers the bug types summarized in Table 17.Table 18 presents the top ten repositories by trajectory count for the main result.Source: §F.3 · p. 42, Table 18 · p. 42, Table 17 · p. 42
  • 58 turns is the average trajectory length, corresponding to 29 actions per demonstration trajectory.Figure 21 visualizes the distribution of trajectory turn counts in the final dataset.Source: §F.3 · p. 42, Figure 21 · p. 42

F.4 Training Analyses … F.5.4 Mitigating repetitive actions

Training analyses report increasing Pass@k and better performance from rejection-sampling fine-tuning, while behavioral studies examine limit sensitivity, command patterns, failure categories, and repetition mitigation that reduces repeated actions without substantially improving success.

F.4 Training Analyses

  • Across six SWE-agent runs, Pass@k increases with k, and fine-tuning on successfully resolved trajectories outperforms random trajectory sampling.The random-sampling ablation used n = [100, 200, 400, 800, 1600] training points and evaluated each student model three times on SWE-bench Verified.Source: §F.4 · p. 42, Figure 22 · p. 43, Figure 23 · p. 43
  • On 300 SWE-bench Multilingual instances, Claude 3.7 Sonnet reached 43% Pass@1, versus SWE-agent-LM-32B’s 8.4% and Qwen 2.5 Coder Instruct’s 6.5%.Spot checks suggested some edits were syntactically closer to Python despite the agent having viewed non-Python code; the authors identify this as a working hypothesis.Source: §F.4 · pp. 42–43

F.5.1 Turn counts and cost

  • Because most failures hit cost or turn limits, average cost and turn counts depend on those limits; SWE-agent-LM-32B resolves more instances under very low step limits.The step-limit chart extrapolates from a run with a 75-step limit, treating a successful run ending at step n as failing under any smaller limit.Source: §F.5.1 · p. 43, Figure 24 · p. 44, Figure 25 · p. 44

F.5.2 Analysis of agent action space

  • To analyze actions, the authors normalize bash commands by stripping leading environment assignments, retaining only the last chained command, and removing arguments while heuristically preserving subcommands.Repetitive actions are defined as the longest sequence of identical base commands, even when editor or view actions target different files.Source: §F.5.2 · pp. 43–44

F.5.3 Failure mode analysis

  • Failure categories first separate errors or context-window overflow from limit-stopped runs, then classify the latter by testing, source edits, and coverage of gold-patch files.Runs with no source edits are distinguished by whether testing was attempted; edited runs are classified by whether all gold-patch source files were modified.Source: §F.5.3 · p. 44
  • Naturally terminated submissions are categorized as incorrect localization or incorrect edit according to whether the submitted patch changes all files in the SWE-bench gold patch.Source: §F.5.3 · p. 44

F.5.4 Mitigating repetitive actions

  • 192 (38.4%) of instances were resolved after repetition mitigation, which reduced repetitive actions but did not substantially improve success rates.The scaffold warns after four or six repeated commands, then resamples actions after longer repetitions, with assistant-message injection or higher temperature at still longer runs.Source: §F.5.4 · p. 45

G Miscellaneous

This section explains how the SWE-smith/SWE-bench scaling comparison was constructed, discusses why web-task trajectory methods transfer poorly to software engineering, and notes the labor still needed for repository setup.

Teaser figure description

  • Figure 1 compares collected SWE-smith instances with estimated SWE-bench yield (20% of candidates; Jimenez et al. (2024b)) across 5–250 repositories sorted by stars.SWE-bench candidates were drawn from pull requests changing Python and testing-related files; repositories were counted at five-repository intervals, starting with those having the fewest stars.Source: G Miscellaneous · pp. 45–46

Extended related works

  • Web-agent trajectory strategies and code-edit-history replay transfer poorly to software engineering, where random command-line interactions rarely produce meaningful codebase effects.Prior web-navigation approaches used random walks labeled with instructions (Xiang et al., 2023; Murty et al., 2024), online tutorials (Ou et al., 2024), or human demonstrations (Shen et al., 2024; Xu et al., 2024); replaying online code-edit sequences also lacked context about skills and decisions.Source: G Miscellaneous · p. 46
  • SWE-smith still requires around 8 minutes of human labor per repository, while end-to-end automation of environment construction remains a goal.The exploration of repository installation and testing specifications was influenced by LM-based environment construction (Bogin et al., 2024) and LM-based unit-test generation (M¨undler et al., 2025).Source: G Miscellaneous · p. 46

Related papers

Loading 2504.21798v2…