PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories

Lilin Wang, Lucas Ramalho, Alan Celestino, Phuc Anthony Pham, Yu Liu, Umang Kumar Sinha, Andres Portillo, Onassis Osunwa, Gabriel Maduekwe

arXiv:2512.17419v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Repository-level benchmarks remain limited by manual curation, narrow language coverage, and difficulty representing feature requests and live software tasks. SWE-Bench++ automates multilingual task generation from GitHub pull requests, and its trajectory data yielded measured gains in a multilingual evaluation. Its execution-based verification remains limited by test-suite quality and does not assess maintainability or algorithmic efficiency.

  • Problem

    Repository-level benchmarks such as SWE-bench rely on manual curation and cover only 12 Python repositories, while automated approaches also face limits in feature-request coverage and heterogeneous test logs.

    Source: §1 · p. 2

  • Method

    SWE-Bench++ automatically turns GitHub pull requests into executable tasks through a four-stage pipeline, with hint-guided trajectory synthesis for instances strong models fail on.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Adding 145 SWE-Bench++ trajectories raised baseline performance from 5/300 to 11/300, while repository-variety gains were statistically significant.

    Source: §4.3.3 · p. 10

  • Takeaways & Limitations

    The framework recovers complex tasks, including feature requests, across heterogeneous build systems that prior methods discard.

    Source: §6 · p. 10

    Execution-based verification is a proxy for correctness, depends on developer test-suite quality, and does not capture maintainability or algorithmic efficiency.

    Source: §5 · p. 10

Abstract

from arXiv · show

Benchmarks like SWE-bench have standardized the evaluation of Large Language Models (LLMs) on repository-level software engineering tasks. However, these efforts remain limited by manual curation, static datasets, and a focus on Python-based bug fixes. We introduce SWE-Bench++, an automated framework that generates repository-level coding tasks from open-source GitHub projects. Unlike synthetic approaches, our pipeline harvests live pull requests to cover both bug fixes and feature requests across 11 languages. SWE-Bench++ turns GitHub pull requests (PRs) into reproducible, execution-based tasks via four stages: programmatic sourcing, environment synthesis, test oracle extraction, and quality assurance. A final hint-guided trajectory synthesis step converts instances that strong models fail on into training trajectories. Our initial benchmark consists of 11,133 instances from 3,971 repositories across 11 languages. On a subset of 1,782 instances of this benchmark, today's strongest models perform as follows: claude-sonnet-4.5 achieves 36.20% pass@10, gpt-5-2025-08-07 34.57%, gemini/gemini-2.5-pro 24.92%, and gpt-4o 16.89%. We further demonstrate the utility of our dataset by showing that fine-tuning on SWE-Bench++ instances yields measurable improvements on the SWE-bench Multilingual benchmark. SWE-Bench++ provides a scalable, multilingual benchmark for evaluating and improving repository-level code generation.

1 INTRODUCTION

Existing repository-level benchmarks are limited by manual curation, narrow language and repository coverage, and difficulty handling feature requests and heterogeneous test logs. SWE-Bench++ proposes an automated multilingual pipeline to generate executable tasks from GitHub pull requests.

1 INTRODUCTION

  • SWE-bench relies on manual curation and covers only 12 Python repositories, a scope described as too small to capture open-source structural and linguistic diversity.The introduction also identifies limitations in synthetic tasks’ representation of noisy, complex, historical human-written code and static benchmarks’ contamination risk.Source: §1 · p. 2
  • Template-guided environment synthesis achieved approximately 137% higher yield than SetUpAgent on the same pool of 2,377 valid pull requests.Adaptive parser synthesis generates custom parsers for heterogeneous logs when deterministic parsers fail.Source: §1 · p. 2
  • Hint-guided trajectory synthesis turns instances that strong models fail on into executable training trajectories, and fine-tuning on 145 trajectories improved SWE-bench Multilingual performance from 1.6% to 3.6%.Function signatures and dependency graphs are injected as hints.Source: §1 · p. 2
  • 11,133 repository-level instances from 3,971 repositories establish the benchmark’s scale, while its automated environment and log-parser synthesis spans 11 languages.The framework also targets diverse build systems and coding patterns.Source: §1 · pp. 2–3
  • The state-differential oracle covers bug fixes and feature requests, expanding feature-like coverage beyond the 9% reported for SWE-bench.It compares Base, Before, and After repository states and treats specific Before-state build failures as signals of feature requests.Source: §1 · pp. 2–3
  • Filtering dated pull requests by creation date enables temporally separated evaluation sets intended to reduce contamination risk for future models.The dataset is constructed from dated GitHub pull requests.Source: §1 · p. 3

2 RELATED WORK

Related work includes manually curated repository benchmarks, synthetic task generation, static data scaling, and live evaluations, each with limits in scope or verification. The section also identifies weaknesses in test oracles and environment reconstruction.

Scalability

  • Synthetic task generation can lack the noise, ambiguity, and in-the-wild distribution of human-written code, while static scraping and trajectory generation remain constrained in execution verification or task diversity.SWE-Fixer prioritizes raw volume without execution environments, and SWE-Gym remains bound to the original manually curated problem set.Source: §2 · p. 3

Data Contamination and “Live” Evaluation

  • Live benchmarks and private-repository evaluation address contamination concerns, but related approaches may remain tied to specific language ecosystems or lack automated verification at thousands-task scale.Static benchmarks are described as vulnerable to memorization of instances created before model knowledge cutoffs.Source: §2 · p. 3

The Weak Test Oracle Problem

  • Passing developer-written tests can admit semantically incorrect or ground-truth-divergent patches, potentially overestimating model capabilities.The section calls for more rigorous state-based verification.Source: §2 · p. 4

Environment Reproducibility Challenges

  • Historical environment reconstruction is a scalability bottleneck because dependency and version mismatches can make configuration errors look like patch failures.Agentic command extraction may be less reliable than deterministic, template-guided scaffolding across polyglot repositories.Source: §2 · p. 4

Solution Leakage and Ambiguity

  • A SWE-bench+ analysis found solution leakage in 32.67% of successful agent resolutions, when issue descriptions or comments contained the correct code or a direct pointer.The section says this can let agents rely on information retrieval rather than reasoning, skewing leaderboard rankings.Source: §2 · p. 4

3 METHODOLOGY: THE SWE-BENCH++ FRAMEWORK

SWE-Bench++ converts GitHub pull requests into executable software-engineering tasks through sourcing, environment synthesis, state-differential test-oracle extraction, and quality assurance. It also applies hint-guided trajectory synthesis to turn model-breaking instances into training data.

3.1 STAGE 1: PROGRAMMATIC SOURCING

  • Candidate pull requests are filtered for active, adopted, complex repositories and merged issue-linked changes that edit or add tests.The coarse filters select repositories with over 100 stars and more than 10k lines of code, among other criteria.Source: §3.1 · p. 4

3.2 STAGE 2: ENVIRONMENT SYNTHESIS

  • Environment synthesis combines structured Dockerfile templates with LLM-inferred dependencies, versions, and entry points.The templates provide a security-conscious structure; repository inspection informs a JSON plan, and build errors feed iterative corrections with up to five retries before a test-run check.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.2 · pp. 4–5

3.2.2 ITERATIVE REFINEMENT WITH BUILD-AND-TEST FEEDBACK

  • Replacing the template-guided design with a SetUpAgent-style baseline reduces Python repository yield to ≈40% of the full system.This comparison quantifies the impact of template-guided environment synthesis.Source: §3.2.2 · p. 5

3.3.1 THREE REPOSITORY STATES

  • The oracle runs tests on Base, Before, and After repository snapshots, with Before containing test changes and After the full pull request.This three-state execution produces distinct logs for automated test-oracle extraction.Source: §3.3.1 · p. 5, §3.3 · p. 5

3.3.2 STATE-DIFFERENTIAL CLASSIFICATION LOGIC

  • State-differential classification treats Before-state build failures tied to missing PR-introduced symbols or dependencies as feature signals, extending fail-to-pass evaluation beyond regression fixes.For buildable Before states, it checks modified tests; for feature requests, newly added tests must pass in After, with the earlier build failure confirming the feature was absent.Source: §3.3.2 · p. 5

3.3.3 SYNTHESIZED ADAPTIVE LOG PARSING

  • Disabling adaptive parser synthesis reduces the final dataset by about 16%, to ≈84% of its original size.The parser strategy uses deterministic parsing first and falls back to LLM-generated parsers for unrecognized formats, checking generated parsers with injected failures.Source: §3.3.2 · p. 5, §3.3.3 · p. 5

3.4 STAGE 4: AUTOMATED INSTANCE QUALITY ASSURANCE & VERIFICATION

  • Automated quality assurance checks build stability and test determinism, assesses problem-oracle alignment, repairs some underspecified tasks with hints, and removes failures traced to dataset or infrastructure defects.Builds and golden-solution tests are repeated three times; SOTA failures are inspected to distinguish model limitations from false negatives.Source: §3.4 · p. 6

3.5 APPLICATION: HINT-GUIDED TRAJECTORY SYNTHESIS FOR TRAINING

  • For model-breaking tasks, hint-guided retries use patch-derived function signatures and dependency graphs, raising pass rates from 0% to ∼70% and yielding training trajectories.A thought-regeneration pass removes hint-related keywords while preserving the logical solution path; the pipeline also includes a manually verified subset of retained model-breaking instances.Source: §3.5 · pp. 6–7

4 EMPIRICAL VALIDATION

SWE-Bench++ is evaluated through pipeline yield and dataset coverage, model baselines, and fine-tuning experiments. Results show gains from diverse SWE-Bench++ trajectories on SWE-bench Multilingual, alongside broad repository and task coverage.

4.1 EVALUATING THE SWE-BENCH++ PIPELINE AND DATASET

  • On the same 2,377 Python pull requests, template-guided synthesis recovered deterministic Docker environments for about 2.37 times as many PRs as SetUpAgent; ablations also found lower yields without template-guided synthesis or adaptive parsing.Without adaptive parsing, the final pipeline retained about 84% of instances; without template-guided synthesis, Python dockerization yield fell to about 40% of the full system’s yield.Source: §4.1 · p. 7
  • The dataset spans 3,971 repositories, with 61.1% of sampled code changes classified as bug fixes and 30.7% as new features; 38.5% of linked issues were feature requests.The issue-level feature-request share exceeds SWE-bench’s approximately 9%.Source: §4.1 · pp. 7–8
  • 28,513 of 137,048 candidate pull requests (20.8%) passed environment synthesis and oracle extraction; quality assurance retained 39% of dockerized instances, yielding 11,133.End-to-end processing averaged around 67 minutes per instance.Source: §4.1 · p. 7, Table 3 · p. 8
  • Task sizes range from 24.5% small changes to 7.6% very large changes, and 17.2% of instances touch 10 or more files.The reported distribution also includes 45.6% medium and 22.3% large changes.Source: §4.1 · p. 8

4.2 EVALUATING LLM AGENTS ON SWE-BENCH++

  • 36.20% for the top-performing model versus low-20s for mid-tier models highlights the challenge of the 1,782-instance benchmark subset.The subset was stratified by language, with roughly 100–280 instances per language.Source: §4.2 · p. 8, Table 4 · p. 9

4.3.1 EXPERIMENTAL SETUP

  • The fine-tuning experiments compare synthetic SWE-Smith data with SWE-Bench++ additions varying in trajectory density, issue diversity, and scale, evaluated on a non-Python multilingual test set without repository overlap.The 200-, 400-, and 800-trajectory scaling mixtures included 40% unreviewed synthetic data; the evaluation set has 300 tasks across 42 repositories and nine languages.Source: §4.3.1 · p. 9

4.3.2 FINE-TUNING EXPERIMENT DETAILS

  • Fine-tuning used a 5e-5 learning rate, up to three epochs, and a 32,768-token context length on eight NVIDIA H200 144G GPUs.The setup followed SWE-Smith’s XML conversion and rejection-sampling fine-tuning process.Source: §4.3.2 · p. 9

4.3.3 RESULTS

  • 145 SWE-Bench++ trajectories (2.8% of the mix) raised performance from 5/300 to 11/300; scaling to 800 trajectories quadrupled the 7B baseline score, while the 32B model peaked at 25/300.The repository-variety improvement was significant (95% CI: +1.0 to +8.0); the solution-density improvement was not distinguishable from baseline (95% CI: +0.0 to +5.0).Source: Table 5 · p. 10, §4.3.3 · p. 10

5 LIMITATIONS AND FUTURE WORK

SWE-Bench++ scales automated execution-based evaluation, but passing tests do not establish maintainability or algorithmic efficiency, and sparse developer tests can admit patches that fail human review.

5 LIMITATIONS AND FUTURE WORK

  • Execution-based verification cannot assess maintainability or algorithmic efficiency, and sparse developer-written tests may let patches pass despite failing human review.The authors identify human-in-the-loop curation and multimodal verification for UI-centric tasks as possible future directions.Source: §5 · p. 10

6 CONCLUSION

SWE-Bench++ combines constrained synthesis, state-differential oracle extraction, and adaptive parsing to recover tasks across heterogeneous build systems, including feature requests. The authors present it as a training resource and living benchmark for multilingual evaluation.

6 CONCLUSION

  • The conclusion presents SWE-Bench++ as an automated, multilingual benchmark that recovers complex tasks, including feature requests, and supplies trajectories that improve performance on an external multilingual benchmark.The paper also describes continuous ingestion of fresh pull requests as a way to minimize contamination risks.Source: §6 · p. 10

A.1 PHASE 1: SAMPLE INPUT AND OUTPUT

  • The build-engineer prompt inspects repositories and dependencies, then produces pre-install, build, test-command, and log-parser configuration for a Docker template.It also directs test-scope filtering and machine-readable or verbose output.Source: §A.1 · p. 13

A.2 PHASE 2: SAMPLE INPUT AND OUTPUT

  • The recovery example replaces a Maven test target that is absent from the reactor with the available project-core and project-api targets.The error log identifies project-aggregator as the missing selected project.Source: §A.2 · p. 14

A.3 PHASE 3: SAMPLE INPUT AND OUTPUT

  • The validator checks that tests started and ran within scope, test names were printed, and failures reflect code or platform issues rather than a broken environment.Failures from missing libraries, toolchain crashes, or harness syntax errors count as environment failures.Source: §A.3 · p. 15

B STANDARD LOG PARSER UTILIZED

  • Standard parsers cover Python test frameworks, JavaScript and TypeScript runners, and Java build tools, including pytest, Jest, Maven Surefire, and Gradle.The appendix also lists parsers for Go, Rust, Ruby, and C/C++.Source: B STANDARD LOG PARSER UTILIZED · p. 16

C.1 QUALITY ANALYSIS METRICS

The appendix defines quality scores for issue clarity and test-to-issue alignment, then reports how automated reviewers compare with human annotators.

C.1 QUALITY ANALYSIS METRICS

  • Issue clarity and test-to-issue alignment each use a 0-to-3 scale, where 0 is best and 3 is worst.Issue clarity considers success criteria, problem specificity, and contextual completeness.Source: §C.1 · p. 16

C.2 QUANTIFYING TEST-TO-ISSUE ALIGNMENT •

  • Test-to-issue alignment assesses core behavior coverage and whether tests falsely reject correct solutions or accept incorrect ones.Poor core coverage and false positives receive higher scores; false negatives are typically addressed by adding cases.Source: §C.2 · p. 16

C.3 PERFORMANCE BY HUMAN VERIFICATION

  • Automated reviewers reach 0.93–0.95 precision versus 0.964/0.963 for human annotators, but GPT-5’s 0.370 recall comes with a 0.154 false-positive rate.Claude-Sonnet-4 recall is 0.921 and Gemini-2.5-Pro recall is 0.884; their higher recall comes with higher false-positive rates relative to humans.Source: §C.3 · p. 16

D.1 PREDICTING WHETHER HINTS ARE NEEDED (I S_H I N T_N E E D E D)

The hint-need module combines build-log parsing, signature checks, and LLM judgment; hint values are kept minimal by combining golden rules with LLM judgment.

D.1 PREDICTING WHETHER HINTS ARE NEEDED (I S_H I N T_N E E D E D)

  • 94.6% accuracy was achieved on 243 senior-labeled instances for predicting whether hints are needed.The module uses build-log failure detection, signature checks via AST or regex, and few-shot LLM judgment.Source: §D.1 · pp. 16–17

D.2 GENERATING HINT VALUES

  • Hint values focus on essentials such as critical signatures, combining golden rules with LLM judgment to avoid biasing context.The stated goal is to maximize fairness while avoiding biasing context.Source: §D.2 · p. 17

E.3 LANGUAGE BREAKDOWN FOR THE 1,782 SAMPLES

The trajectory-generation scaffold uses SWE-Agent with multiple LLMs to increase trajectory diversity, since different models induce distinct solution paths.

E.3 LANGUAGE BREAKDOWN FOR THE 1,782 SAMPLES

  • Multiple LLMs with SWE-Agent are used to increase trajectory diversity because different models induce distinct solution paths.Curation selects SOTA-model-breaking issues and uses tailored Docker environments produced earlier in the pipeline.Source: §F.1 · pp. 17–18

F.2 TRAJECTORY GENERATION AND SELECTION

The paper generates successful agent trajectories for model-breaking GitHub issues using tailored hints, then applies automated quality checks and human review.

F.2 TRAJECTORY GENERATION AND SELECTION

  • Hints and multiple attempts raise the trajectory passing rate from approximately 0% to ∼70% in this setting.Success requires passing both pass-to-pass and fail-to-pass tests.Source: §F.2 · p. 18
  • The curated trajectories solve real GitHub issues spanning bug fixes, feature requests, and refactoring, rather than synthetic constraints focused mainly on regression bugs.The pipeline harvests trajectories for SOTA model-breaking issues at scale without relying on pre-existing model solutions.Source: §F.2 · p. 18
  • Thought, action, and observation sequences capture iterative codebase exploration and provide richer supervision than single-shot solutions or static agentless flows.The trajectories are intended for standard fine-tuning regimes such as SFT and DPO.Source: §F.2 · p. 18

F.3 AUTOMATED TRAJECTORY QUALITY ASSURANCE

  • Automated QA removes structurally invalid or hint-contaminated traces, regenerates affected thoughts, and judges retained trajectories for quality.The evaluator scores problem reproduction for bug fixes, solution plausibility, validation evidence, and engineering practices.Source: §F.3 · pp. 18–19

F.4 HUMAN REVIEW

  • Human reviewers inspect trajectories that pass automated thresholds for hint over-reliance and failure modes, and may revise or discard them.This review targets quality aspects that algorithmic checks cannot reliably handle.Source: §F.4 · p. 19
  • The resulting HITL-augmented process yields agentic demonstrations designed to fine-tune code-capable language models.The trajectories encode both reasoning and interaction dynamics.Source: §F.4 · p. 19

G FINE-TUNING EXPERIMENT DETAILS IN §4.3

Fine-tuning follows the SWE-Smith setup, with three training epochs, an effective batch size of eight, and a maximum sequence length of 32,768.

G FINE-TUNING EXPERIMENT DETAILS IN §4.3

H CASE STUDIES IN FAILURE ANALYSIS OF SOTA MODELS

The two cases show SOTA models failing through mistaken assumptions about token representation and multiline matching. The framework’s diagnoses point to raw-buffer inspection in yamllint and whole-content regex matching with newline mapping in changedetection.io.

H.1 CASE 1 — YAMLLINT: MISINTERPRETING RAW VS. PREPROCESSED TOKENS

  • In yamllint, quotes are needed for a double-quoted multiline string when a line ends with a backslash, which removes the next line’s leading spaces.When lines break on spaces without a trailing backslash, the example says quotes are unnecessary.Source: §H.1 · p. 20
  • The framework identifies yamllint’s raw token buffers as preserving backslashes, contrary to the model’s assumption that tokens were preprocessed.A helper inspects raw buffers for backslashes at line endings.Source: §H.1 · p. 20

H.2 CASE 2 — CHANGEDETECTION.IO: MISHANDLING MULTILINE REGEX AND ZERO-WIDTH MATCHES

  • changedetection.io’s per-line processing and the model’s span-overlap heuristics failed on a zero-width multiline-regex unit test.The test used /^$/ms$.Source: §H.2 · pp. 20–21
  • Whole-content regex matching with newline mapping fixes multiline matching when zero-width spans defeat overlap checks.The approach evaluates matches over the full content, maps match starts to lines by counting newlines, and avoids overlap heuristics.Source: §H.2 · p. 21
Loading 2512.17419v1…