PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes

Silin Chen, Zihan Zhang, Xiaodong Gu, Jiale Huang, Yuling Shi, Han Li, Zhao Shan, Chengcheng Wan, Zhensu Sun, Haibing Guan

arXiv:2610.21359v1cs.SE

TL;DR

Real-world SWE tasks are constrained by the pace of issue reporting and the cost of curation, while prompting-based synthesis can revisit familiar bug patterns. SWE-at-Scale learns bug-inducing transformations by reversing real fixes, and its generated tasks are realistic, diverse, and challenging, with bug diversity continuing to rise through 200 generations per file.

  • Problem

    Real-world SWE tasks are difficult to scale because issue reporting, repository availability, and executable-environment construction constrain their collection.

    Source: §1 · p. 2

  • Method

    SWE-at-Scale reverses real software fixes into bug-inducing supervision and learns reusable transformations for synthesizing tasks in new code contexts.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Human evaluators favored SWE-at-Scale- and SWE-at-Scale+-generated bugs as more likely to occur in real software development in 55.9% and 61.2% of cases, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Bug-type diversity continues to increase with sampling, with different bug types generated from the same code context and no full saturation within 200 generations per file.

    Source: §6 · p. 10

    The study focuses exclusively on Python repositories, so generalization to other programming languages remains unclear.

    Source: §6 · p. 9

Abstract

from arXiv · show

As coding agents become increasingly capable, robust evaluation requires a broader and more diverse supply of verifiable software engineering (SWE) tasks. However, real-world tasks are constrained by the natural frequency of issue reporting and the substantial cost of manual curation. Furthermore, existing synthetic approaches rely primarily on the pretrained knowledge of large models coupled with heuristic prompting, often failing to capture the authentic, complex mechanics of software failures. To exponentially scale SWE task production, we introduce SWE-at-Scale, a framework that automatically synthesizes realistic issues by reverse-engineering real-world software fixes. Our key insight is that historical fixes encapsulate deep mechanistic knowledge of bug formation: inverting a fix transforms repair supervision into bug-induction supervision grounded in actual development scenarios. SWE-at-Scale transfers these learned bug patterns to novel code contexts, repeatedly sampling diverse bug typologies to construct executable SWE tasks. Extensive experiments show that SWE-at-Scale generates more diverse and realistic SWE tasks than current synthesis baselines while posing greater challenges to state-of-the-art coding agents. Notably, SWE-at-Scale can generate distinct bug types from a single code context, with bug diversity scaling robustly even up to 200 generations per file. Overall, SWE-at-Scale turns finite real-world fixes into reusable supervision for scalable synthesis of realistic, diverse, and challenging SWE tasks.

1 INTRODUCTION

Real-world SWE tasks are costly to scale, while prompting general-purpose models can revisit familiar bug patterns. SWE-at-Scale learns bug-inducing transformations from reversed fixes and applies them to new code contexts, producing realistic, diverse, and challenging tasks.

1 INTRODUCTION

  • Real-world SWE tasks scale slowly because issue reporting, repository availability, and executable-environment construction constrain their collection.Scaling solely through naturally occurring issues also requires continually discovering and validating additional development events.Source: §1 · p. 2
  • Prompt-based synthesis relies on general LLMs’ pretrained bug-generation knowledge, which can lead repeated generations to revisit familiar patterns.SWE-at-Scale instead reverses real fixes into supervision for learning reusable bug-inducing transformations.Source: §1 · p. 2
  • SWE-at-Scale trains on 13,407 real-fix transformations from 688 repositories and provides direct-patch and natural-language-pattern variants.The framework uses reversed fixes to learn real-world bug patterns that can be instantiated in new code contexts.Source: §1 · pp. 2–3
  • Human evaluators favored SWE-at-Scale- and SWE-at-Scale+-generated bugs as more likely to occur in real software development in 55.9% and 61.2% of cases, respectively.Across four resolution models, Pass@1 was 31.68%–48.48% for SWE-at-Scale and 16.81%–37.02% for SWE-at-Scale+; bug types were more balanced than in existing synthesis methods.Source: Figure 1 · p. 1, §1 · pp. 2–3
  • Even after 200 generations per file, 88.27% of usable patches remain unique, while tasks remain challenging for current coding agents.SWE-at-Scale+ approaches the difficulty of human-expert-designed tasks such as DeepSWE.Source: §1 · p. 3

2 SWE-AT-SCALE: SOFTWARE TASK SYNTHESIS FROM REAL FIXES

SWE-at-Scale reverses historical fixes into correct-to-buggy supervision, trains models to induce bugs in new code, and validates resulting repository-level tasks through tests and issue descriptions.

2 SWE-AT-SCALE: SOFTWARE TASK SYNTHESIS FROM REAL FIXES

  • SWE-at-Scale reverses historical fixes into (golden code, buggy code) pairs, then trains a model to generate buggy code from source files.The framework applies the learned transformations to new code contexts to synthesize task instances.Source: §2 · p. 3

2.1 TRAINING DATA COLLECTION

  • The training corpus contains 13,407 real-fix transformations from 688 repositories, selected after filtering, reconstruction, distribution-aware sampling, and manual inspection.The balancing strategy broadens coverage across bug types, repositories, and software domains while retaining the prevalence of Logic and Interface transformations in real-fix data.Source: §2.1 · p. 4

2.2 TRAINING BUG-INDUCING MODEL

  • SWE-at-Scale fine-tunes Qwen3.5-9B to generate a bug-inducing diff from a complete golden source file, learning both where and how to modify code.Providing the complete file lets the model infer edit locations from surrounding context.Source: §2.2 · p. 4
  • SWE-at-Scale+ predicts an executor-ready natural-language bug pattern that specifies the target, modification, constraints, and expected effect before code changes are realized.This variant makes the intended bug transformation explicit rather than leaving the pattern implicit in a generated patch.Source: §2.2 · p. 4

2.3 TASK ASSEMBLY AND VALIDATION

  • Task assembly retains generated changes only when execution makes at least one previously passing test fail, then pairs the validated repository with an issue description.The issue description is generated from the diff, a Pass-to-Fail test, its execution output, and demonstrations.Source: §2.3 · p. 5

3 EXPERIMENTS

The experiments compare SWE-at-Scale with synthetic and real-world task sources under controlled synthesis and model-disjoint resolution settings. They assess realism, usability, and difficulty using human judgments, execution filters, and downstream resolution success.

3.1 SETUP

  • Baselines include SWE-smith’s LLM-Modify and LLM-Rewrite, BugPilot’s FeatAdd, and real-world SWE-bench Verified and DeepSWE issues.Methods synthesize from the same target file and golden repository state, with up to five attempts per file.Source: §3.1 · p. 5
  • Downstream evaluation uses model-disjoint issue resolution, three independent runs, and a maximum of 250 actions per task.Each model resolves tasks whose problem statements were not generated by that same model.Source: §3.1 · p. 5

3.2 EVALUATION METRICS

  • Realism is measured by engineers’ pairwise human preference, usability by the share of target commits yielding a validated non-trivial task, and difficulty by Pass@1.Usability requires a previously passing regression test to fail; Table 1 reports downstream issue-resolution performance across task instances.Source: §3.2 · pp. 5–6, Table 1 · p. 6

4 RESULTS

SWE-at-Scale produces usable, realistic tasks that remain challenging, while repository coverage and inference-time sampling expand within-file bug diversity. Larger sampling budgets continue to add diversity, and the method produces larger bug-inducing patches at lower inference cost than rewrite-based synthesis.

4.1 SYNTHESIS QUALITY AND TASK DIFFICULTY

  • SWE-at-Scale+ reaches 94.0% Usability Rate and 61.2% HPR, versus 72.2% and 51.8% for the best LLM-Rewrite configuration.SWE-at-Scale reaches 72.6% usability and 55.9% HPR; Figure 5 compares the two measures.Source: §4.1 · p. 6, Figure 5 · p. 6
  • Across evaluated resolution models, both SWE-at-Scale variants yield lower Pass@1 than SWE-bench Verified and synthetic baselines, with SWE-at-Scale+ lowest in every setting.The difficulty trend generalizes across solvers, and SWE-at-Scale tasks approach the difficulty of human-expert-designed DeepSWE tasks.Source: §4.1 · p. 7

4.2 SCALING BUG DIVERSITY AT TRAINING AND INFERENCE TIME

  • Mean Bug Type Entropy measures the average diversity of usable bug types generated for the same target file.It computes entropy over each file’s usable candidates and macro-averages across target files.Source: §4.2 · p. 7
  • Increasing training repositories from 5 to 100 raises MBTE while usability remains relatively stable and improves modestly for the largest pools.The comparison holds training data size fixed at 2,000 datapoints.Source: §4.2 · pp. 7–8, Figure 6 · p. 7
  • MBTE peaks at 1.173 at temperature 0.6, while usability falls from 69.0% at 0.0 to 23.4% at 1.0.Temperatures 0.4–0.6 provide the most favorable diversity–usability trade-off; higher temperatures reduce both diversity and usability.Source: Figure 7 · p. 7, §4.2 · p. 8
  • At K = 200, SWE-at-Scale reaches 2.063 bits of MBTE while usability stabilizes near 47%; 4,192 of 4,749 usable candidates (88.27%) remain unique.Diversity continues to rise through K = 200, although growth begins to converge around K = 100.Source: §4.2 · p. 8, Figure 8 · p. 8
  • At K = 200, LLM-Modify with DS-V4-Flash reaches 1.22 bits of MBTE and 44.1% usability, with 61.74% of usable candidates unique.SWE-at-Scale reaches 2.063 bits of MBTE, about 47% usability, and an 88.27% unique-candidate rate at the same budget.Source: §4.2 · p. 8

4.3 BUG COMPLEXITY AND SYNTHESIS COST

  • SWE-at-Scale and SWE-at-Scale+ modify 18.793 and 18.536 lines on average, versus 14.334 for SWE-bench Verified.The direct path uses 17.45k tokens per instance on average, less than LLM-Rewrite’s 34.92–62.82k and far below FeatAdd’s 2.29M.Source: §4.3 · p. 8

5 RELATED WORK

Real-world SWE benchmarks and synthetic-task approaches address software-engineering evaluation at scale, but naturally occurring tasks remain bounded by development and curation constraints. SWE-at-Scale differs by turning real fixes into reusable training supervision for generating new tasks across code contexts.

5.1 REAL-WORLD SOFTWARE ENGINEERING BENCHMARKS

  • Naturally occurring SWE task collections are limited by software-development pace, repository availability, and the costs of authoring and validating tasks.The related benchmarks include SWE-bench, continuously extracted SWE-rebench tasks, enterprise-oriented SWE-Bench Pro, long-horizon DeepSWE tasks, and refactoring-focused SWE-Bench ProMax.Source: §5.1 · p. 9

5.2 SYNTHETIC SOFTWARE ENGINEERING TASKS

  • Synthetic-task methods include code rewriting and execution filtering, feature implementation with retained regressions, and transferring issue patterns to similar repositories.SWE-smith, SWE-Synth, BugPilot, and SWE-Mirror exemplify these approaches, respectively.Source: §5.2 · p. 9
  • Unlike prompting or source-issue transfer, SWE-at-Scale trains on real fixes to learn reusable bug-inducing patterns that can be instantiated across new code contexts.The paper presents this as a way to expand the diversity of synthesized SWE tasks.Source: §5.2 · p. 9

6 DISCUSSION

The discussion identifies language coverage and test-suite coverage as boundaries on the tasks SWE-at-Scale can construct. It also summarizes the framework’s findings and provides AI-use and reproducibility statements.

6 DISCUSSION

  • The study is limited to Python repositories, and generalization to other languages remains unclear; bugs missed by tests may be discarded, especially in repositories with weak test coverage.Source: §6 · p. 9
  • Bug-type diversity continues to increase as the sampling budget grows, without fully saturating within 200 generations per file.Source: §6 · p. 10

AI USE STATEMENT

  • AI tools assisted with manuscript grammar checking and code for experiment data processing and visualization.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

A.1 LORA FINE-TUNING CONFIGURATION

The appendix reports the LoRA fine-tuning configuration used for both SWE-at-Scale variants, with Table 2 covering the backbone, optimization hyperparameters, and training settings.

A.1 LORA FINE-TUNING CONFIGURATION

  • Table 2 summarizes the backbone, optimization hyperparameters, and training settings used in all experiments.Source: §A.1 · p. 15
  • The reported configuration is for LoRA fine-tuning of both SWE-at-Scale and SWE-at-Scale+.Source: §A.1 · p. 15, Table 2 · p. 15
  • The table presents the LoRA fine-tuning configuration used for the two synthesis variants.Source: Table 2 · p. 15

A.2 BUG COMPLEXITY AND SYNTHESIS COST

This appendix figure reports two measures of synthesis cost: mean changed lines per instance and average total tokens consumed to produce one instance.

A.2 BUG COMPLEXITY AND SYNTHESIS COST

  • Figure 9 reports mean changed lines per synthesized instance and average total token consumption per instance.Source: §A.2 · p. 15, Figure 9 · p. 15
  • The token-consumption measure is the average total required to produce one synthesized instance.Source: §A.2 · p. 15
  • The changed-line measure is averaged across synthesized instances and evaluated synthesis configurations.Source: Figure 9 · p. 15, §A.2 · p. 15

Related papers

Loading 2610.21359v1…