PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Software Engineering Agents and Verifiers with SWE-Gym

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, Yizhe Zhang

arXiv:2412.21139v2cs.SEcs.CL

TL;DR

Software-engineering agents lack realistic training environments with executable feedback, limiting efforts to improve the underlying language models. SWE-Gym supplies real-world Python tasks, environments, and tests for training agents and verifiers, achieving 32.0% on SWE-Bench Verified and 26.0% on SWE-Bench Lite with inference-time scaling.

  • Problem

    Software engineering lacks suitable training environments with executable runtimes, reproducible tests, and step-by-step interaction or reward signals.

  • Method

    SWE-Gym provides 2,438 real-world Python tasks with repository context, pre-installed dependencies, executable environments, and expert-validated tests for training agents and verifiers.

  • Results

    32.0% on SWE-Bench Verified and 26.0% on SWE-Bench Lite are achieved when SWE-Gym-trained agents use verifiers for inference-time trajectory selection.

  • Takeaways & Limitations

    SWE-Gym supports scalable improvements for open-weight software-engineering agents and verifiers as training and inference compute increase.

  • Takeaways & Limitations

    SWE-Gym remains limited in environment diversity across repositories, task types, and programming languages.

Abstract

from arXiv · show

We present SWE-Gym, the first environment for training real-world software engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task instances, each comprising a codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, achieving up to 19% absolute gains in resolve rate on the popular SWE-Bench Verified and Lite test sets. We also experiment with inference-time scaling through verifiers trained on agent trajectories sampled from SWE-Gym. When combined with our fine-tuned SWE agents, we achieve 32.0% and 26.0% on SWE-Bench Verified and Lite, respectively, reflecting a new state-of-the-art for open-weight SWE agents. To facilitate further research, we publicly release SWE-Gym, models, and agent trajectories.

1. Introduction

SWE-Gym addresses the lack of realistic, executable training environments for software engineering agents. Training agents and verifiers on its tasks improves benchmark resolution and continues to benefit from additional compute.

  • Motivation: Existing SWE-agent progress relies heavily on proprietary models, while software engineering lacks suitable training environments for improving the underlying language model.Real-world SWE requires executable runtimes, dependencies, reproducible tests, and interaction signals that existing resources often omit.
  • Contribution: SWE-Gym combines real-world GitHub issues, pre-installed dependencies, and executable test verification across 2,438 Python tasks from 11 open-source repositories.The environment is designed for training both language-model agents and verifiers.
  • Agent Training: +12.3% to 15.3% on SWE-Bench Lite and +13.6% to 20.6% on SWE-Bench Verified are achieved by fine-tuning a 32B Qwen-2.5 coder model on 491 SWE-Gym trajectories.The agent uses the OpenHands scaffold and only the sampled SWE-Gym interaction trajectories for fine-tuning.
  • Agent Training: Self-improvement with SWE-Gym boosts performance up to 19.7% on SWE-Bench Lite across a specialized agent scaffold.The method uses rejection-sampling fine-tuning while the language model interacts with SWE-Gym and receives reward from it.
  • Verifier Training: 32.0% on SWE-Bench Verified and 26.0% on SWE-Bench Lite result when a SWE-Gym-trained verifier selects among sampled agent trajectories.The verifier estimates each trajectory’s probability of success, enabling inference-time scaling and a new state-of-the-art among publicly weighted systems.
  • Scaling: Performance improves continuously with increasing compute during both training and inference, with no saturation observed through 491 training trajectories.The observed scaling suggests current training performance is limited by trajectory-sampling compute rather than the number of SWE-Gym tasks.

2. Related Work

Prior software-engineering resources and agent designs leave important gaps in realism, executable feedback, and trainability. SWE-Gym is positioned as a publicly available environment that supplies real repository-level tasks and test-based reward signals.

  • Software Engineering Agents: SWE agents solve GitHub issues by modifying associated repositories and producing code patches verified against human-written test suites.Agent scaffolds range from specialized workflows with human-defined stages to general-purpose prompting over longer interaction histories.
  • Software Engineering Agents: Specialized workflows reduce task horizons but require substantial human engineering and may fail on intermediate steps, whereas general-purpose prompting is more flexible but computationally expensive.General-purpose approaches also demand higher capabilities from the underlying language model.
  • Training Environments: SWE-Bench training lacks executable environments and success signals, while HumanEval targets standalone coding and R2E uses synthetic tasks.These resources therefore do not provide a suitable dataset for training agents on realistic repository-level software engineering.
  • Training Environments: SWE-Gym combines real GitHub tasks, repository-level context, pre-installed dependencies, executable environments, and expert-written tests as reward signals.This directly addresses the training-environment gap that has encouraged reliance on prompting and proprietary models.

3. SWE-Gym Environment

SWE-Gym is constructed from large-scale GitHub issue data and narrowed to validated executable instances. Its repository distribution is long-tailed, and its tasks are generally harder than SWE-Bench tasks.

  • Dataset Construction: SWE-Gym comprises 2,438 validated tasks from 11 Python repositories, while SWE-Gym Raw contains 64,689 issues across 358 repositories.SWE-Gym Lite contributes a 230-instance subset for faster prototyping.
  • Dataset Construction: Repositories are filtered using age, popularity, code-size, pull-request, and contributor thresholds before task instances are extracted from GitHub issues.Each extracted instance includes an issue description, repository snapshot, and unit tests.
  • Environment Setup: SWE-Gym Raw lacks executable environments and guarantees that unit tests effectively evaluate correctness, so 11 repositories receive semi-manual environment construction.Version-specific setup helps avoid redundant environment work across instances.
  • Environment Setup: Gold patches are validated by checking that they pass more unit tests than the original code, yielding 2,438 instances after filtering.Construction required approximately 200 human annotation hours and 10,000 CPU core hours, with reproducible Docker images released for each instance.
  • Dataset Analysis: SWE-Gym’s repository distribution is long-tailed: pandas contributes nearly one-third of tasks, whereas bokeh contributes about one percent.The distribution indicates that instances are unevenly concentrated across repositories.
  • Dataset Analysis: SWE-Gym tasks are on average harder than SWE-Bench tasks, with larger gold patches and consistently lower model performance.The difficulty difference may reflect inclusion of sophisticated repositories such as pandas and MONAI, in addition to possible benchmark overfitting.

4. Training LMs as Agents with SWE-Gym

SWE-Gym trains open-weight SWE agents using executable real-world tasks and sampled success trajectories. Fine-tuning improves benchmark resolution and reduces looping, while specialized workflows enable limited self-improvement.

  • 4.1. Setting: SWE-Gym experiments use OpenHands and MoatlessTools to evaluate training across general-purpose prompting and specialized workflows.The evaluation reports resolution rate, Empty Patch, and related agent behaviors on SWE-Bench Lite and Verified.
  • 4.2. Training General-Purpose Prompting Agents: Rejection sampling fine-tuning trains the base language model on successful trajectories sampled from SWE-Gym.The collected training set contains 491 successful trajectories, primarily limited by computational budget.
  • 4.2. Training General-Purpose Prompting Agents: Fine-tuning generally reduces stuck-in-loop behavior by 4.6–18.6%, alongside lower empty-patch rates, with one 32B Lite exception increasing by 1.5%.The 32B model on SWE-Bench Lite already had a low loop rate before training.
  • 4.2. Training General-Purpose Prompting Agents: Larger base models consistently improve resolution, Empty Patch, and stuck-in-loop rates.The passage reports this pattern across the evaluated model sizes and benchmarks.
  • 4.3. Self-Improvement with Specialized Workflow: MoatlessTools reaches 7% and 19% zero-shot resolution for 7B and 32B models, versus 1.0% and 3.0% with OpenHands on SWE-Bench Lite.Its specialized workflows constrain the action space and reduce task horizons.
  • 4.3. Self-Improvement with Specialized Workflow: Per-instance capping addresses easy-data bias by limiting successful trajectories per task; a threshold of two slightly outperforms the full dataset and speeds training.Low caps reduce dataset size and performance, while high caps skew training toward easier tasks.
  • 4.3. Self-Improvement with Specialized Workflow: After two rejection-sampling iterations, the 7B model rises from 7.0% to 10.0%, while the 32B model rises from 19.0% to 19.7% with no further gains.The authors attribute the limited 32B gains to the restricted action space and rejection-sampling method.

5. Scaling Agent Performance with SWE-Gym

SWE-Gym enables scaling through learned verifiers and additional training trajectories. More inference compute and sampled training data improve performance, while verifier quality and data composition remain important.

  • 5.1. Scaling Agent Performance with SWE-Gym: SWE-Gym supports inference-time scaling by training outcome-supervised reward models to rerank sampled agent trajectories.The verifier estimates trajectory success from task execution context and selects the highest-reward candidate.
  • 5.1.1. VERIFIER FOR GENERAL-PURPOSE PROMPTING: Pass@k rises from 20.6% at k=1 to 37.8% at k=8 and 42.8% at k=16 on SWE-Bench Verified.Pass@k measures whether at least one sampled trajectory succeeds.
  • 5.1.1. VERIFIER FOR GENERAL-PURPOSE PROMPTING: Best@k rises from 20.6% at k=1 to 29.8% at k=8 and 32.0% at k=16, indicating steady but smaller gains from verifier selection.The gap from Pass@k reflects imperfect verifier performance.
  • 5.1.1. VERIFIER FOR GENERAL-PURPOSE PROMPTING: A mixture of on-policy and off-policy verifier data reaches 27% resolution at k=8, outperforming on-policy-only and off-policy-only alternatives.The ablation reports off-policy-only performance plateauing around 22%.
  • 5.1.2. VERIFIER FOR SPECIALIZED WORKFLOW: On SWE-Bench Lite, a 7B verifier raises a 7B policy from 10% to 13.3%, while a 32B verifier raises a 32B policy from 19.7% to 26.3%.The 7B verifier plateaus after k=4, whereas the 32B verifier continues improving through k=8.
  • 5.2. Scaling Training Data: Randomly increasing training trajectories consistently improves resolution, particularly for the 32B model, suggesting current task and repository diversity are not bottlenecks.The scaling study uses 491 sampled trajectories and compares trajectory, task-instance, and repository scaling.
  • 5.2. Scaling Training Data: Among deduplicated approaches, repository scaling performs better initially at 25% data, while random no-deduplication achieves higher final performance partly because it uses more trajectories.The comparison reports 491 trajectories for random scaling versus 294 after deduplication.

6. Conclusions, Limitations, and Future Work

SWE-Gym combines real-world Python tasks, repository context, executable environments, and test verification to support scalable training of software engineering agents. The paper identifies environment diversity, limited reinforcement learning progress, and absent human-in-the-loop collaboration as important directions for future work.

  • Conclusions: SWE-Gym combines real-world Python tasks with repository-level context, pre-configured execution environments, and test verification for scalable software engineering agent training.
  • Limitations and Future Work: Environment diversity remains limited across repositories, task types, and programming languages, motivating automated environment, test-case, and task synthesis.
  • Limitations and Future Work: Self-improvement results are modest, making large-scale online reinforcement learning a proposed direction for further improvement.
  • Limitations and Future Work: Current software engineering settings focus on task completion and neglect human-in-the-loop collaboration needed for real-world software engineering.

Impact Statement

The paper discusses potential benefits of automated software engineering for productivity and accessibility alongside risks involving employment, code governance, and misuse. It calls for responsible deployment frameworks addressing security, safety, and economic impacts.

  • Improved automated software engineering could increase developer productivity and accessibility, although current models are research artifacts rather than production-ready systems.
  • More capable agents may affect software engineering jobs and raise questions about code ownership, licensing, and attribution.
  • Similar techniques could potentially automate malicious-code creation, motivating responsible deployment frameworks for security, safety, and economic impacts.

A. Comparison with Concurrent Works

Concurrent work comparison emphasizes differences in feedback, environment construction, analysis scope, compute, model size, and openness. SWE-Gym provides executable feedback with manually configured environments and publicly released project artifacts.

  • Lingma SWE-GPT uses manually defined heuristics without executable unit-test feedback, and its training pipeline and dataset are not public.
  • Concurrent executable environments use a general setup script and discard failed instances, biasing datasets away from tasks with complex dependencies.
  • SWE-Gym provides executable feedback, manually configured environments, comprehensive agent and verifier analyses, and positive self-improvement results.
  • SWE-Gym achieves competitive results with significantly lower compute and a smaller model size, 32B versus 72B.
  • Table 5 compares SWE-Bench model performance with whether model weights and environments are publicly accessible.
  • Table 6 reports resolution rate and empty patch rate for a 7B model trained with different instance-capping strategies.

B. Experiment Details

The experiment estimates uncertainty for Pass@N and Best@N curves by repeatedly subsampling rollout sets. A special handling of N=1 applies to the OpenHands CodeActAgent inference-time scaling curve because its first attempt uses temperature 0.

  • For N < M, mean and variance are computed across 100 random sub-samples of size N from M rollout rounds.
  • The OpenHands CodeActAgent curve excludes the N=1 calculation because its first attempt uses temperature 0.

B.2. OpenHands Agent Experiments

OpenHands experiments use full-parameter fine-tuning across 7B, 14B, and 32B models, with parallel trajectory execution and bounded evaluation. Sampling analyses examine success distributions and training-data scaling under different trajectory-selection strategies.

  • Training setup: OpenHands trajectories are executed in parallel on SWE-Gym’s remote runtime during training.Training uses full-parameter fine-tuning with a 1e-4 learning rate, up to 5 epochs, batch size 8, and a 32768-token context limit.
  • Training setup: 7B, 14B, and 32B model variants are fine-tuned, with evaluation bounded by 100 interaction turns or the 32k context window.The bound uses whichever limit is reached first.
  • Sampling analysis: Training-time scaling compares deduplicated trajectories, unique task instances, and repositories as alternative sampling approaches.The comparison uses a 32B language model.
  • Sampling analysis: Per-instance capping reduces the easy-task bias in 30-round success distributions but lowers the total trajectory count available for training.The SWE-Gym Lite experiment uses a 7B model in zero-shot sampling at temperature t = 1.
  • Verifier setup: The verifier-training exception uses LoRA for two epochs, producing 29.8% @8 in the main experiment.Other hyperparameters remain unchanged from the stated training configuration.

B.3. MoatlessTools Agent Experiments

MoatlessTools experiments use distinct fine-tuning configurations for 7B and 32B models, while serving MoatlessAgent experiments with FP8 quantization for throughput.

  • 7B configuration: The 7B MoatlessTools policy model uses full fine-tuning on four H100 GPUs for five epochs with batch size 8 and learning rate 2 × 10^-5.All MoatlessTools models use a 10240-token context window.
  • 32B configuration: The 32B model uses LoRA fine-tuning for five epochs with batch size 8, rank 64, and learning rate 5 × 10^-4.The same configuration is used for verifier training, with a single H100 GPU.
  • Serving: MoatlessAgent experiments use FP8 quantization to improve throughput, with minimal observed effects on model performance.

B.4. Details of OpenHands Trajectory Sampling

OpenHands trajectory sampling builds training data from successful SWE-Gym Lite runs across temperatures, while verifier prompts evaluate trajectories or patches and support inference-time selection. The experiments distinguish solution discovery from verifier-based ranking and compare several data mixtures and scaling strategies.

  • Trajectory sampling: Sampling SWE-Gym Lite with GPT-4o and temperatures from 0.2 to 0.8 produces successive sets of successful trajectories for fine-tuning.The initial temperature-0 set contains 19 successful trajectories, and the combined D1 set contains 106.
  • Trajectory sampling: 491 successful trajectories form the final D2 training set.The total is assembled as 106 + 19 + 67 + 299 success trajectories.
  • Verifier prompts: The MoatlessTools verifier prompt uses trajectory-derived context rather than proprietary-model context extraction.The prompt is modified from prior work and obtains context directly from the agent trajectory being evaluated.
  • Verifier prompts: The OpenHands verifier receives an interaction log and judges whether the assistant successfully resolved the user’s request against task, edge-case, error, and verification criteria.The training prompt includes a judgement, while inference predicts the probability of Yes and No.
  • Evaluation: Verifier-training ablations compare mixtures of off-policy and on-policy data, including doubled negative examples from Claude and GPT.The reported comparison uses Best@k.
Loading 2412.21139v2…