PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan

arXiv:2310.06770v3cs.CLcs.AIcs.SE

TL;DR

Existing benchmarks do not adequately capture the frontier of language-model capabilities in real-world software engineering. SWE-bench evaluates issue resolution on GitHub-derived tasks, finding that models solve only the simplest issues and that Claude 2 resolves 1.96% of them.

  • Problem

    Existing language-model benchmarks have become saturated and fail to capture the frontier of real-world capabilities.

    Source: §1 · p. 1

  • Method

    SWE-bench evaluates models by applying generated patches for GitHub issues to Python repositories and checking whether associated tests pass.

    Source: §1 · p. 2, §2 · p. 2, §2.2 · p. 3

  • Results

    1.96%: Claude 2 resolves only this share of issues, while models generally solve only the simplest cases.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    SWE-bench provides a realistic, execution-based benchmark for assessing language models on complex software-engineering issues.

    Source: §1 · p. 2

    SWE-bench task instances are limited to Python, with expansion to other programming languages and domains left for future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

1 INTRODUCTION

SWE-bench addresses saturated benchmarks by evaluating language models on realistic GitHub software-engineering issues, where generated patches are tested against repository code. It provides a diverse, continuously extensible benchmark and accompanying training resources, but Claude 2 resolves only 1.96% of issues with BM25 retrieval.

  • SWE-bench responds to saturated existing benchmarks by targeting challenging, verifiable real-world applications of language models.Coding tasks remain attractive because generated solutions can be verified by running unit tests.Source: §1 · pp. 1–2
  • SWE-bench evaluates patches for real GitHub issues by applying them to repository snapshots and running the repositories’ testing frameworks.Tasks typically involve bug reports or feature requests and may require changes across functions and files.Source: Figure 1 · p. 1, §1 · p. 2
  • The benchmark offers realistic user-submitted tasks, problems from 12 repositories, execution-based evaluation, and continuous updates with minimal human intervention.These properties distinguish it from coding benchmarks such as HumanEval, which mostly use self-contained problems solvable in a few lines.Source: §1 · p. 2
  • 1.96% of SWE-bench issues are resolved by Claude 2 with BM25 retrieval, indicating that models fail on all but the simplest tasks.The evaluation covers multiple state-of-the-art language models.Source: §1 · p. 2
  • SWE-bench-train contains 19,000 non-testing instances from 37 repositories and supports SWE-Llama 7b and 13b, with SWE-Llama 13b competitive with Claude 2 in some settings.SWE-Llama 13b can process contexts exceeding 100,000 tokens.Source: §1 · p. 2

2 SWE-BENCH

SWE-bench evaluates repository-scale issue resolution by requiring models to edit real Python codebases and pass associated tests. Its filtered, continually extensible design captures realistic, diverse, cross-context software engineering tasks, with a 300-instance Lite subset for easier adoption.

2.1 BENCHMARK CONSTRUCTION

  • SWE-bench comprises 2,294 task instances filtered from about 90,000 pull requests across 12 popular open-source Python repositories.The three-stage pipeline selects issue-resolving PRs with test changes, retains fail-to-pass tests, and excludes installation or runtime errors.Source: §2.1 · pp. 2–3

2.2 TASK FORMULATION

  • Models receive an issue description and complete codebase, generate a patch, and succeed only when it applies and all associated unit and system tests pass.The benchmark metric is the percentage of task instances resolved.Source: §2.2 · p. 3

2.3 FEATURES OF SWE-BENCH

  • SWE-bench targets real-world software engineering through large codebases, long issue descriptions, and unrestricted edits spanning multiple locations.Reference solutions average editing 1.7 files, 3.0 functions, and 32.8 lines, while issue descriptions average 195 words and codebases contain thousands of files.Source: §2.3 · p. 3
  • The benchmark supports continual extension to Python repositories with minimal human intervention, enabling evaluation on issues created after models’ training dates.Its repository-scale setting also supports comparisons among retrieval, long-context, and agentic approaches while allowing solutions to diverge from reference pull requests.Source: §2.3 · p. 3

2.4 SWE-BENCH LITE

  • SWE-bench Lite provides a 300-instance subset sampled for greater self-containment and functional bug-fix evaluation, covering 11 of the original 12 repositories.The subset maintains similar repository diversity and distribution while reducing the burden of adopting the full benchmark.Source: §2.4 · pp. 3–4

3 SWE-LLAMA: FINE-TUNING CODELLAMA FOR SWE-BENCH

SWE-Llama fine-tunes 7B- and 13B-parameter CodeLlama-Python models into repository editors capable of generating repository-wide fixes for GitHub issues. Training uses 19,000 issue–PR pairs from repositories disjoint from SWE-bench, with LoRA and token filtering reducing the effective corpus to 10,000 instances.

  • Fine-tuning 7B- and 13B-parameter CodeLlama-Python models produces specialized repository editors that can run on consumer hardware and resolve GitHub issues.Off-the-shelf CodeLlama variants typically produced placeholders or unrelated code instead of following detailed repository-wide editing instructions.Source: §3 · p. 4

Training data

  • 19,000 issue–PR pairs from 37 additional Python repositories form the training set, whose repositories are disjoint from the evaluation benchmark to prevent contamination.Unlike the benchmark data collection procedure, the training set does not require pull requests to include test changes, enabling a larger supervised fine-tuning corpus.Source: §3 · p. 4

Training details

  • LoRA fine-tunes only attention-sublayer weights, while excluding sequences longer than 30,000 tokens reduces the effective training corpus to 10,000 instances.The prompt contains the issue text, instructions, and relevant code files; the target is the gold patch that solved the issue.Source: §3 · p. 4

4 EXPERIMENTAL SETUP

SWE-bench evaluation combines issue descriptions with large codebases, requiring retrieval to select model context. The setup compares retrieval conditions, constructs patch-generation inputs, and evaluates four long-context-capable models.

4.1 RETRIEVAL-BASED APPROACH

  • SWE-bench instances pair short issue descriptions averaging 195 words with codebases averaging 438K lines, exceeding typical language-model context windows.This motivates selecting relevant code context before evaluation.Source: §4.1 · p. 4
  • At the 27,000-token limit, BM25 retrieves a superset of oracle files in approximately 40% of instances but retrieves none of them in almost half.BM25 uses generic sparse retrieval, while oracle retrieval supplies files edited by the reference patch and is less realistic and potentially incomplete.Source: §4.1 · p. 5, Table 3 · p. 5

4.2 INPUT FORMAT

  • The model input combines task instructions, issue text, retrieved files, documentation, an example patch, and a prompt to generate the patch.Appendix D provides examples and further formulation details.Source: §4.2 · p. 5

4.3 MODELS

  • The evaluation covers ChatGPT-3.5, GPT-4, Claude 2, and SWE-Llama because only a few models support the long sequences required by SWE-bench.Their context limits are reported in Table 4, whose token lengths are non-standard across tokenizers; Llama sequences average 42% longer than equivalent GPT-4 sequences.Source: §4.3 · p. 5, Table 4 · p. 5

5 RESULTS

Models resolve very few SWE-bench issues, with performance constrained by retrieval, context length, and context-distribution shifts. Qualitative analysis shows that generated patches are often narrow, primitive, and less structurally robust than gold solutions.

5 RESULTS

  • 1.96% is the highest BM25-retrieval performance, achieved by Claude 2, while models broadly struggle to resolve SWE-bench issues.With oracle retrieval, Claude 2 resolves 4.8% of issues.Source: §5 · p. 6
  • Oracle-retrieval difficulty varies across repositories, and models solve substantially different issue sets: Claude 2 solves only 42% of instances solved by SWE-Llama 13b.In the oracle setting, Claude 2 and SWE-Llama 13b resolve 110 and 91 instances, respectively.Source: §5 · p. 6
  • Claude 2’s performance drops considerably as total context length increases, indicating difficulty localizing problematic code despite greater retrieval recall.Collapsing oracle context to edited lines plus a ±15-line buffer raises GPT-4 from 1.3% to 3.4% and Claude 2 from 4.8% to 5.9%.Source: §5 · pp. 6–7
  • Finetuned SWE-Llama models perform poorly with BM25 context because training used oracle-retrieval inputs, creating a context-distribution shift.BM25 includes files that are not expected to change, unlike the finetuning setup where models were trained to edit every contextual file.Source: §5 · p. 7
  • Models generate shorter, simpler patches than gold solutions, with successful edits averaging 30.1 lines versus 74.5 gold lines and rarely spanning multiple files.Generating entire files performs worse than generating patches: Claude 2 scores 2.2% versus 4.8% with oracle retrieval.Source: §5 · p. 7

5.1 A QUALITATIVE ANALYSIS OF SWE-LLAMA GENERATIONS

  • Qualitative analysis finds that models use primitive, greedy solutions with limited reuse of libraries or codebase patterns, whereas gold patches make broader structural improvements.In one Sphinx example, SWE-Llama edited the correct function but ignored the napoleon.use_param configuration, causing tests to fail.Source: §5.1 · pp. 8–9

6 RELATED WORK

SWE-bench addresses limitations in existing language-model evaluations, code-generation benchmarks, and software-engineering datasets by preserving realistic, broad, and challenging software-development tasks.

Evaluation of LMs

  • SWE-bench targets the simplicity and narrowness of “potpourri” evaluations with nuanced, practical, refreshable tasks that support deeper insight into model capabilities.The benchmark is presented as significantly challenging, with a wide range of possibilities for improving language models.Source: §6 · p. 9

Code Generation Benchmarks

  • SWE-bench preserves real-world engineering challenges beyond closed-form completion, including patch generation, long-context reasoning, codebase navigation, and dependency relationships across modules.Its collection procedure applies minimal post-processing rather than partitioning and simplifying problems into siloed datasets.Source: §6 · p. 9

ML for Software Engineering

  • Unlike existing software-engineering datasets, SWE-bench provides code context at a larger scale and supports more realistic experiments with software-engineering tools and practices.The benchmark can also be extended to new programming languages and repositories.Source: §6 · p. 9

7 DISCUSSION

SWE-bench mirrors real-world coding environments through open-source collaboration, while highlighting limitations in language coverage, methodology, and execution-based evaluation. The authors position it as a resource for developing more practical, intelligent, and autonomous language models.

Limitations and future directions

  • The benchmark currently covers only Python, motivating expansion to additional programming languages and domains.The authors hope to apply SWE-bench’s task-instance collection procedure beyond its current scope.Source: §7 · p. 10
  • Execution-based testing alone cannot guarantee reliable model performance because generated code may be less comprehensive, efficient, or readable than human-written solutions.The authors also encourage future work on methods beyond the simplest baseline approaches, including agent-based and tool-augmented language models.Source: §7 · p. 10

Conclusion

  • SWE-bench creates a faithful mirror of real-world coding environments by drawing on the open-source collaborative pipeline.The benchmark is intended to encourage creative solutions with immediate applicability to open-source software development.Source: §7 · p. 10

8 ETHICS STATEMENT

SWE-bench uses publicly licensed repositories without collecting GitHub user information or involving human participants, and it plans an open, documented release with feedback channels while noting no immediately harmful insights.

Ethics and data practices

  • The benchmark uses publicly licensed repositories, relies only on public GitHub data, collects no user information, and involves no human subjects or crowdsourced workers.Repository selection by popularity does not use discriminative or biased heuristics.Source: §8 · p. 11

Release and impact

  • The planned release includes task instances, collection and evaluation infrastructure, experimental results, SWE-Llama training data and weights, documentation, and feedback channels.The paper states that SWE-bench does not provide any immediately harmful insights and discusses potential usage impacts in Section E.Source: §8 · p. 11

9 REPRODUCIBILITY STATEMENT … A.4 EVALUATION PROCEDURE

The appendix documents a reproducible, extensible SWE-bench pipeline that constructs task instances from GitHub pull requests, validates them through execution, and evaluates model-generated patches against cached test outcomes. It also releases the code, task instances, mirrors, and documentation needed to reproduce the benchmark.

9 REPRODUCIBILITY STATEMENT

  • SWE-bench releases anonymized source code, all 2,294 task instances, documentation, and collection infrastructure, supporting reproducibility and planned open-source maintenance.The codebase separates dataset collection, evaluation, inference, and training components, while Sections A.2 and A.4 provide additional technical details.Source: §9 · p. 11

APPENDIX

  • The appendix expands the main paper with technical details on dataset construction, execution-based validation, evaluation, and benchmark characterization.These materials complement Section 2 with a finer-grained account of the benchmark’s data and procedures.Source: APPENDIX · p. 16, A BENCHMARK DETAILS · p. 16

A BENCHMARK DETAILS

  • The collection framework selects licensed, widely used Python repositories and converts merged pull requests that resolve issues and add tests into candidate task instances.The framework is extensible to new repositories, languages, and domains, and can generate temporally updated instances from pull requests created after a model’s training date.Source: §A.1 · p. 16

A.1 HIGH LEVEL OVERVIEW

  • Each task instance combines an issue-derived problem statement, a mirrored repository at its base commit, test patch T, and gold solution patch δ, while excluding later comments that could leak solutions.Repository mirrors preserve original commit history and hashes, and patches are separated by testing-related file paths.Source: §A.1 · p. 16, §A.2 · pp. 17–18

A.2 CONSTRUCTION PROCESS

  • Task construction records issue, repository, patch, timestamp, and version metadata, with version information supporting temporal analyses and executable-context selection.Mirrors make base-commit retrieval reliable, while parsed test and solution patches are checked by applying them and reproducing the pull request’s behavior.Source: §A.2 · pp. 17–19

A.3 EXECUTION-BASED VALIDATION

  • Execution validation uses release-version environments to install each codebase, run tests before and after applying δ, and retain only nontrivial instances with at least one fail-to-pass test.Candidates are discarded when installation, patching, or testing fails; instances with problematic pre-solution import or attribute errors are also removed, and cached status transitions support later evaluation.Source: §A.3 · pp. 19–20

A.4 EVALUATION PROCEDURE

  • Evaluation gives the codebase and problem statement to a language model, applies its predicted patch with optional automatic repair, runs the tests, and scores success only when every FAIL TO PASS and PASS TO PASS test passes.A failed repair or test run receives score 0, while successful outputs are converted into repository-specific test-status mappings for comparison with cached ground truth.Source: §A.4 · pp. 20–21

A.5 EVALUATION TEST SET CHARACTERIZATION … B ADDITIONAL DETAILS ON TRAINING SWE-LLAMA

SWE-bench task instances span diverse issue types and typically involve concise descriptions, large codebases, small reference patches, and tests for both correctness and behavioral preservation. The development set follows the same construction process, while SWE-bench Lite is introduced as a more efficient canonical subset.

A.5 EVALUATION TEST SET CHARACTERIZATION

  • SWE-bench adds repository-specific statistics for edited lines and pass-to-pass tests, which check that revisions preserve existing behavior.Fail-to-fail and pass-to-fail tests are excluded from evaluation statistics.Source: §A.5 · p. 21, Table 11 · p. 23
  • SWE-bench issues are mostly bug fixes but cover diverse code changes beyond debugging and error correction.The categorization analyzes metadata tags and identifies 2,289 tags across the task instances.Source: §A.5 · p. 21, Table 13 · p. 24
  • A median SWE-bench task has a 140-word description in a codebase with nearly 1,900 files and 400K lines, while its reference solution usually edits one function.The typical patch changes approximately 15 lines and includes one fail-to-pass test plus 51 pass-to-pass tests.Source: §A.5 · pp. 21, 23
  • SWE-Llama patches require post-generation fixing less often among successfully applied patches, whereas closed-source models apply fewer patches and need more fixes.These patterns suggest differences in patch formatting and structured-output reliability.Source: §A.5 · p. 23, Table 14 · p. 24

A.6 DEVELOPMENT SET CHARACTERIZATION

  • The development set contains 225 tasks from six repositories and uses the main evaluation set’s methodologies and filters, with an added post-2019 creation-date filter.Its repositories have licenses permitting the stated development usage.Source: §A.6 · p. 24, Table 15 · p. 25
  • Development tasks retain diverse coverage beyond bug fixing and include repository-specific categories such as crashes and I/O problems.The development set therefore includes issue types not present in the main evaluation tasks’ tag examples.Source: §A.6 · p. 24, Table 16 · p. 25
  • Across development tasks, each instance averages 19.9 fail-to-pass tests, 171.3 pass-to-pass tests, and 191.2 tests overall.The corresponding medians are 2, 79.0, and 101.0 tests, respectively.Source: §A.6 · p. 24

A.7 SWE-BENCH LITE CHARACTERIZATION

  • SWE-bench Lite is introduced as a canonical subset for more efficient language-model evaluation on SWE-bench.The supplied passage begins its characterization but does not provide further subset statistics.Source: §A.7 · p. 25

B.1 TRAINING DETAILS … C.4 EXTENDED TEMPORAL ANALYSIS

The appendix details LoRA training for SWE-Llama and extends evaluation across oracle retrieval, repositories, a GPT-4 subset, and task creation years. These analyses show improved performance with cleaner retrieval, strong repository variation, representative GPT-4 subset results, and no consistent temporal performance trend.

B.1 TRAINING DETAILS

  • SWE-Llama is trained with LoRA on attention projections using r = 16, α = 16, dropout = 0.05, learning rate 6e −4, batch size 32, and at most 4 epochs.The best checkpoint is selected by validation loss on 100 held-out instances; 7b and 13b models train in 20 and 47 hours on 4 and 8 NVIDIA A100s, respectively, using DeepSpeed Ulysses and Flash Attention for long contexts.Source: §B.1 · p. 25

C.1 RESULTS WITH “ORACLE” RETRIEVAL

  • Oracle retrieval improves model performance over the noisier BM25 retrieval setting by providing only files edited in the reference solution.The comparison uses the oracle retrieval method described in Section 4.1 and is reported in Table 18.Source: §C.1 · p. 25, Table 18 · p. 26

C.2 EVALUATION TEST SET

  • Model performance differs heavily across repositories, with repository-level results reported alongside repository-grouped SWE-bench task statistics.The performance breakdown corresponds to Figure 4 in the main paper and is reported in Table 19, while Table 17 summarizes task-instance attributes by repository.Source: §C.2 · p. 25, Table 17 · p. 26

C ADDITIONAL RESULTS

  • Table 18 compares models under oracle retrieval, while GPT-4 is evaluated on a 25% random SWE-bench subset in the oracle and BM25 27K retriever settings because of budget constraints.The oracle results are contrasted with the main Table 5 results, which use BM25 only.Source: Table 18 · p. 26

C.3 GPT-4 EVALUATION SUBSET RESULTS

  • The GPT-4 25% random evaluation subset has % Resolved and % Apply rates consistent with the main results and is not significantly skewed toward easier or harder instances.The subset was evaluated under the settings summarized in Table 20.Source: §C.3 · p. 26

C.4 EXTENDED TEMPORAL ANALYSIS

  • Across six temporal partitions under oracle retrieval, model % Resolved performance shows no consistent correlation with the year issues were created.This supports the conclusion that SWE-bench fixes require understanding and cannot feasibly or consistently be solved through memorization of observed data.Source: §C.4 · p. 26

C.5 F2P, P2P RATE ANALYSIS · C.6 PATCH GENERATION EXTENDED ANALYSIS · C.7 SOFTWARE ENGINEERING METRICS

Extended analyses show that most successfully applied but unresolved patches are No-Ops or regressions, while longer patches remain difficult to format correctly. Software engineering metrics further expose quality differences that line counts alone miss.

C.5 F2P, P2P RATE ANALYSIS

  • The six outcome categories distinguish whether successfully applied patches satisfy desired behavior and preserve prior behavior, including Resolved, Breaking Resolved, Partially Resolved, Work in Progress, No-Op, and Regression.The categories are defined by F2P and P2P pass or fail results.Source: §C.5 · pp. 26–27, Table 22 · p. 28
  • Among successfully applied patch generations for non-Resolved issues, most do not solve any corresponding F2P test; among these, 60% to 70% are No-Op, and the remainder break existing behavior.Source: §C.5 · p. 27
  • Partial outcomes are less common, and failures often arise because baseline methods miss inter-file dependencies and relationships among affected functions.The authors suggest execution-environment feedback could let models test fixes, continue editing, or submit patches more reliably.Source: §C.5 · pp. 27–28

C.6 PATCH GENERATION EXTENDED ANALYSIS

  • Across models, generated patches resemble average gold edits in size, but longer correctly formatted output sequences remain difficult to produce.Longer generations commonly exhibit hallucinations, violations of existing code style or structure, and incorrect handling of long-range dependencies.Source: §C.6 · p. 28, Table 24 · p. 29

C.7 SOFTWARE ENGINEERING METRICS

  • Software engineering metrics offer scalar, static-analysis feedback on patch complexity, efficiency, readability, and effects across complex codebases.The analysis uses Radon to compute metrics directly from source code, focusing on Cyclomatic and Halstead complexity.Source: §C.7 · pp. 28–29
  • A six-line Claude 2 patch solved the issue but raised HTTPAdapter Cyclomatic complexity from 3 to 5, whereas the eleven-line gold patch raised simpler functions from 2 to 3 and 0 to 1.The model changed one file and inserted a conditional into a widely used class; the reference changed two files and introduced dependencies and a new error type.Source: §C.7 · p. 29, Figure 10 · p. 30

D ADDITIONAL EXPERIMENTAL DETAILS … E SOCIETAL IMPACT

The experimental details specify filename-aware sparse retrieval, oracle path extraction, greedy single-patch inference, and a prompt that supplies the issue, codebase, and patch format. The paper also positions SWE-bench as a testbed for safe, aligned, and verifiable AI-driven software engineering.

D.1 RETRIEVAL DETAILS

  • Filename-aware sparse retrieval prepends file contents with paths, while oracle retrieval extracts reference-patch paths excluding tests.Source: §D.1 · p. 30

D.2 INFERENCE SETTINGS

  • Each instance receives one generated patch, using greedy decoding for all models in a Pass@1-style evaluation.Generations are described as relatively expensive, motivating the single-patch setting.Source: §D.2 · p. 30

D.3 PROMPT TEMPLATE EXAMPLE

  • The prompt presents a partial codebase and issue statement, with repository files delimited inside a code block.The template includes README.md and Python-file contents alongside the issue text.Source: §D.3 · pp. 30–31
  • The requested output is a single git-applyable patch file describing filenames, line numbers, and removed or added lines across potentially multiple files.The template provides an example patch and explicitly asks for one patch file in the shown format.Source: §D.3 · pp. 31–32
  • Varying the number of instructional or example lines generally did not substantially affect performance, apart from findings reported in Section 5.Source: §D.3 · p. 32

E SOCIETAL IMPACT

  • SWE-bench is proposed as a controlled testbed for developing safeguards that make AI-driven software engineering aligned, verifiable, robust, and safe.The motivation centers on preserving human intent and containing harms when code agents misinterpret objectives.Source: E SOCIETAL IMPACT · p. 32

F IN-DEPTH ANALYSIS OF SWE-LLAMA GENERATIONS

This section provides an in-depth analysis of Claude 2 and SWE-Llama generations, examining successful and failed patches across issues of varying complexity. It highlights the importance of codebase understanding and identifies challenges involving multi-file changes, repository-specific reasoning, dependencies, and other requirements beyond local edits.

Related papers

Loading 2310.06770v3…