PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan

arXiv:2310.06770v3cs.CLcs.AIcs.SE

TL;DR

Existing benchmarks do not adequately capture the frontier of language-model capabilities in real-world software engineering. SWE-bench evaluates issue resolution in realistic repositories, finding that models solve only the simplest issues, with Claude 2 resolving 1.96% of tasks.

  • Problem

    Existing language-model benchmarks have become saturated and fail to capture the frontier of capabilities needed for real-world applications.

  • Method

    SWE-bench evaluates language models by applying generated patches for GitHub issues to Python repositories and checking whether associated tests pass.

  • Results

    Models struggle across SWE-bench, solving only the simplest issues; using BM25 retrieval, Claude 2 resolves 1.96% of tasks.

  • Takeaways & Limitations

    SWE-bench provides a realistic benchmark for developing more practical, intelligent, and autonomous language models.

  • Takeaways & Limitations

    SWE-bench currently covers only Python, and the experiments establish baselines using straightforward approaches rather than constraining future methods.

Abstract

from arXiv · show

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

1 INTRODUCTION

SWE-bench addresses saturated benchmarks with a realistic, execution-verified evaluation of language models on GitHub software-engineering issues. State-of-the-art models solve only the simplest issues, while the released training data and SWE-Llama models support further development.

  • Benchmark design: SWE-bench evaluates models by generating patches for real GitHub issues in popular Python repositories and testing them against related tests.Its task setup connects GitHub issues to merged pull-request solutions and provides an issue description with a codebase snapshot.
  • Benchmark design: The benchmark captures software engineering challenges involving repository navigation, cross-file function interplay, and errors in convoluted code.These issues differ from self-contained coding tasks such as HumanEval, which can usually be solved in a few lines.
  • Benchmark advantages: SWE-bench combines user-submitted issues and solutions, diverse problems from 12 repositories, execution-based evaluation, and continuous updates requiring minimal human intervention.These properties are presented as advantages over existing language-model programming benchmarks.
  • Evaluation results: 1.96%: Claude 2 resolves only this share of SWE-bench issues using a BM25 retriever, with models otherwise failing on all but the simplest issues.The evaluation covers multiple state-of-the-art language models.
  • Additional contributions: 19,000 non-testing task instances: SWE-bench-train derives this collection from 37 repositories and supports two fine-tuned models, SWE-Llama 7b and 13b.The authors describe the training dataset as essential for advancing open model development and report that SWE-Llama 13b is competitive with Claude 2 in some settings.

2 SWE-BENCH

SWE-bench is a repository-scale benchmark built from filtered GitHub issues and pull requests, requiring models to edit complete codebases and pass associated tests. Its realistic, continually extensible design emphasizes long-context, cross-file software engineering, while a 300-instance Lite subset supports easier adoption.

  • Construction: SWE-bench contains 2,294 task instances filtered from about 90,000 pull requests across 12 popular open-source Python repositories.Tasks require merged pull requests that resolve issues, modify tests, include fail-to-pass tests, and avoid installation or runtime errors.
  • Task and evaluation: A model receives an issue description and complete codebase, generates a patch, and succeeds only when the patch applies and all associated unit and system tests pass.Benchmark performance is the percentage of task instances resolved.
  • Benchmark properties: SWE-bench targets real-world software engineering through large codebases, detailed issue descriptions averaging 195 words, and revisions spanning multiple locations.Codebases regularly contain many thousands of files, requiring models to identify the relatively small number of lines relevant to an issue.
  • Benchmark properties: Its collection process can continually add post-training-date issues from Python repositories with minimal human intervention, enabling evaluation without relying on solutions in training data.SWE-bench also supports comparing retrieval, long-context, and decisionmaking-agent approaches while allowing solutions to differ from reference pull requests.
  • Lite subset: The 300-instance SWE-bench Lite subset emphasizes self-contained functional bug fixes and covers 11 of the original 12 repositories.It was created to encourage adoption while preserving similar repository diversity and distribution.

3 SWE-LLAMA: FINE-TUNING CODELLAMA FOR SWE-BENCH

SWE-Llama was developed to evaluate open-model performance on SWE-bench because off-the-shelf CodeLlama models could not reliably follow repository-wide editing instructions. It was fine-tuned with LoRA on issue–code prompts and gold patches using a contamination-controlled corpus of 10,000 effective training instances.

  • Motivation: Off-the-shelf CodeLlama variants typically produced placeholders or unrelated code instead of repository-wide edits from detailed instructions.CodeLlama was identified as the only model family able to handle SWE-bench’s very long contexts at the time.
  • Training data: 19,000 issue-PR pairs were collected from 37 additional Python repositories, disjoint from evaluation repositories to prevent data contamination.Unlike the benchmark collection procedure, training pairs did not require pull requests to include test changes, enabling a larger supervised fine-tuning set.
  • Training details: SWE-Llama was fine-tuned to generate gold patches from GitHub issue text and relevant code files, updating only attention-sublayer weights with LoRA.Sequences longer than 30,000 tokens were excluded, reducing the effective training corpus to 10,000 instances.

4 EXPERIMENTAL SETUP

SWE-bench evaluation provides models with issue descriptions, codebases, retrieved context, documentation, and patch-generation instructions. The setup compares BM25 sparse retrieval with an oracle file-based setting and evaluates four models suited to long sequences.

  • Inputs and context: SWE-bench instances pair an issue description with a codebase, averaging 195 words and 438K lines respectively.The codebases typically exceed language-model context windows, creating a context-selection problem.
  • Context retrieval: The baselines use BM25 sparse retrieval or oracle retrieval, with sparse retrieval tested under three maximum context limits.BM25 selects as many relevant files as fit the specified limit, while oracle retrieval uses files edited by the reference patch.
  • Context retrieval: At the 27,000-token limit, BM25 retrieves a superset of oracle files in approximately 40% of instances but retrieves none of them in almost half.The oracle setting is less realistic and may still omit context needed to understand software behavior.
  • Input construction: The model input combines task instructions, issue text, retrieved files, documentation, an example patch, and a patch-generation prompt.Further formulation details and examples are provided in Appendix D.
  • Evaluated models: The evaluation covers ChatGPT-3.5, GPT-4, Claude 2, and SWE-Llama because few models currently support the required long sequence lengths.Their context limits are reported in Table 4.

5 RESULTS

Models struggle to resolve SWE-bench issues, with performance limited by retrieval quality, context length, repository variation, and context-distribution shifts. Their patches are often shorter and simpler than gold edits, reflecting limited use of surrounding code and coding conventions.

  • Overall performance: 1.96%: Claude 2 resolves the highest share of issues under BM25 retrieval, while models broadly struggle on the benchmark.With oracle retrieval, Claude 2 resolves 4.8% of issues.
  • Difficulty analysis: Repository difficulty trends similarly across models, but their solved instances overlap incompletely: Claude 2 solves 42% of SWE-Llama’s oracle-resolved instances.In the oracle setting, Claude 2 resolves 110 instances and SWE-Llama 13b resolves 91.
  • Difficulty analysis: 32%: matplotlib instances contain embedded images, compared with 2% of all instances, potentially requiring multimodal models or external image-processing tools.Seaborn contains embedded images in 10% of instances.
  • Context and retrieval: As total context length increases, Claude 2’s performance drops considerably, a pattern also observed in other models.The evaluation supplies substantial code that may not directly relate to resolving the issue.
  • Context and retrieval: 4.8% to 5.9%: Claude 2 improves when oracle-retrieved files are collapsed to edited lines plus ±15 lines of buffer.GPT-4 similarly rises from 1.3% to 3.4%, indicating that irrelevant context can hinder performance.
  • Difficulty analysis: Finetuned SWE-Llama 7b and 13b perform surprisingly poorly with BM25 context after training on oracle-retrieved context.The suspected distribution shift is that BM25 includes files not expected to be changed, whereas training taught SWE-Llama to edit every contextual file.
  • Patch characteristics: 74.5 versus 30.1 lines: correctly applied model patches are less than half the length of average gold patches and rarely edit more than one file.Models tend to produce shorter, simpler edits than the corresponding gold patches.
  • Patch characteristics: Models tend to write primitive, greedy solutions that underuse third-party libraries, surrounding code, code style, and logical constraints.A documented example edits the correct function but fails tests because it compares output under the wrong configuration.

6 RELATED WORK

Related work evaluates language models across diverse tasks and interactive web settings, benchmarks code generation beyond HumanEval, and applies neural models to automate software engineering processes.

  • Evaluation of LMs: LM evaluation benchmarks span mutually distinct tasks across domains and interactive web settings requiring multiple solution steps.The passage notes drawbacks to this “potpourri” setup but truncates before specifying them.
  • Code Generation Benchmarks: HumanEval remains the standard for synthesizing code from natural-language descriptions, while newer benchmarks extend it across languages and edit scopes.The passage identifies subsequent benchmarks as extensions to different languages and variations in edit scope.
  • ML for Software Engineering: Neural networks, including language models, are being used to automate real-world software development processes that traditional program analysis may not scale to or incorporate natural language into.Reported use cases include commit generation and pull-request review.

7 DISCUSSION

SWE-bench reflects the complexity of real-world software development through an open-source collaborative pipeline, while highlighting limitations in language coverage, methodology, and execution-based evaluation. The authors present it as a foundation for future LM development and broader benchmark coverage.

  • Limitations and future directions: The benchmark currently covers only Python, motivating expansion to additional programming languages and domains.The authors hope to apply SWE-bench’s task-instance collection procedure beyond Python.
  • Limitations and future directions: Execution-based code testing alone cannot guarantee reliable model performance because generated code may be less comprehensive, efficient, or readable than human-written solutions.The authors therefore identify evaluation beyond automated testing as an important limitation of the current approach.
  • Conclusion: SWE-bench mirrors real-world coding environments by drawing on the open-source collaborative pipeline.The benchmark is intended to encourage creative solutions with immediate applicability to open-source software development.

8 ETHICS STATEMENT

SWE-bench uses publicly available, appropriately licensed repository data and avoids collecting GitHub user information or involving human subjects.

  • Data and participation: SWE-bench draws entirely from public repositories whose licenses permit software use, with license details provided in Table 12.The task uses GitHub data only through the public API and website.
  • Data and participation: The collection and evaluation processes collect no GitHub user information and involve no human subject participation or crowdsourcing.

9 REPRODUCIBILITY STATEMENT … A.4 EVALUATION PROCEDURE

The paper releases anonymized, documented code and fully specified SWE-bench artifacts, while detailing reproducible task construction, execution-based validation, and prediction evaluation procedures. The benchmark is designed for extensibility, temporal robustness, and reliable assessment of whether model patches reproduce intended test behavior.

  • 9 REPRODUCIBILITY STATEMENT: The submission includes anonymized source code, organized by contribution, with inline documentation and the full set of 2,294 task instances.The released code covers dataset collection, evaluation, open-source model inference, and SWE-Llama training.
  • APPENDIX: The appendix expands the paper’s benchmark description with technical details on dataset construction, evaluation, validation, and task-instance characterization.
  • A.1 HIGH LEVEL OVERVIEW: SWE-bench converts merged, issue-resolving pull requests from popular licensed Python repositories into task instances containing a codebase, problem statement, tests, and solution.Repositories are selected from highly downloaded PyPI packages, and task codebases are identified by repository owner/name and pull-request base commit.
  • A.1 HIGH LEVEL OVERVIEW: The collection process supports low-maintenance extension to new repositories, programming languages, and domains, while enabling updates with pull requests created after a model’s training date.
  • A.2 CONSTRUCTION PROCESS: Task-instance construction derives problem statements from issue and pull-request text, retrieves codebases through repository mirrors and base commits, and extracts solution and test patches from pull-request diffs.Additional metadata includes creation timestamps for temporal analysis and repository release versions for executable contexts.
  • A.3 EXECUTION-BASED VALIDATION: Validation creates release-version executable contexts, installs each base codebase, applies tests and solutions, and discards candidates when any required execution step fails.Candidates are also filtered for problematic pre-solution errors and must contain at least one test transitioning from fail to pass.
  • A.4 EVALUATION PROCEDURE: Evaluation resets the repository, activates its versioned environment, installs the codebase, applies tests and the model-generated patch, optionally repairs patch formatting, and runs the tests.A failed patch application or test command receives a score of 0.
  • A.4 EVALUATION PROCEDURE: Task completion requires every ground-truth FAIL TO PASS and PASS TO PASS test to appear with pass status in the model’s evaluation results.

A.5 EVALUATION TEST SET CHARACTERIZATION … B ADDITIONAL DETAILS ON TRAINING SWE-LLAMA

The supplemental sections characterize SWE-bench’s evaluation and development sets through repository, issue-type, task-attribute, and patch-generation analyses, and introduce SWE-bench Lite as a more efficient evaluation subset. They also report development-set testing statistics and describe repository licensing and task diversity.

  • A.5 EVALUATION TEST SET CHARACTERIZATION: SWE-bench provides repository-specific attribute statistics and repository descriptions with associated open-source licenses permitting non-commercial source-code use under their original terms.The expanded statistics add δ # Lines Added, δ # Lines Removed, and |T| (Pass to Pass).
  • A.5 EVALUATION TEST SET CHARACTERIZATION: 2,289 issue tags show that SWE-bench tasks cover diverse contribution types, although bug fixes constitute the absolute majority.The analyses characterize issue contributions using metadata, particularly tags associated with pull requests.
  • A.5 EVALUATION TEST SET CHARACTERIZATION: The median SWE-bench task has a 140-word description, nearly 1,900 files, and 400K lines of code.Reference solutions usually edit one function, changing approximately 15 lines, and use one fail-to-pass test plus 51 pass-to-pass tests.
  • A.5 EVALUATION TEST SET CHARACTERIZATION: Patch-fixing procedures are analyzed across models for generated, successfully applied, and post-generation-fixed patches over 2,294 task instances.The text reports that fixed patches make up a smaller percentage of successfully applied SWE-Llama patches, suggesting a positive effect from fine-tuning.
  • A.6 DEVELOPMENT SET CHARACTERIZATION: The development set contains 225 tasks from 6 open-source repositories and follows the same collection methodology as the main evaluation set.Its repository licenses permit the described usage, and it is intended for evaluation and hyperparameter tuning before final testing.
  • A.6 DEVELOPMENT SET CHARACTERIZATION: Development tasks include diverse issue tags beyond bug fixing, including repository-specific categories such as “Crash :collision:” and “io”.These tags indicate problems unique to repositories represented in the development set.
  • A.6 DEVELOPMENT SET CHARACTERIZATION: Development tasks have 19.9 average / 2 median F2P tests, 171.3 average / 79.0 median P2P tests, and 191.2 average / 101.0 median total tests.These figures summarize testing coverage across the entire development set.
  • A.7 SWE-BENCH LITE CHARACTERIZATION: SWE-bench Lite is a canonical subset intended to enable more efficient evaluation of language models on SWE-bench tasks.The supplied passage introduces the subset but does not provide its size or selection criteria.

B.1 TRAINING DETAILS … C.4 EXTENDED TEMPORAL ANALYSIS

The appendix details LoRA training and reports additional SWE-bench analyses using oracle retrieval, repository-level breakdowns, a GPT-4 subset, and temporal partitions. Oracle retrieval improves performance over noisier BM25 retrieval, while performance varies substantially across repositories and shows no consistent correlation with issue year.

  • B.1 TRAINING DETAILS: LoRA fine-tuning used r = 16, α = 16, dropout = 0.05, a learning rate of 6e −4, batches of 32 sequences, and a maximum of 4 epochs.Checkpoints were saved every 50 steps and selected using validation loss on 100 held-out instances.
  • C.1 RESULTS WITH “ORACLE” RETRIEVAL: Oracle retrieval improves model performance compared with the noisier BM25 retrieval setting by providing only files edited by the reference pull request.These general performance results are reported in Table 18.
  • C.2 EVALUATION TEST SET: Model performance differs heavily across repositories in the repository-by-repository breakdown corresponding to Figure 4.The breakdown is reported in Table 19.
  • C.2 EVALUATION TEST SET: Table 17 reports average task-instance attributes grouped by repository for repositories in the development dataset.It reproduces the same statistics presented in Table 11.
  • C.3 GPT-4 EVALUATION SUBSET RESULTS: GPT-4’s 25% random evaluation subset has % Resolved and % Apply rates consistent with the main results and is not significantly skewed toward easier or harder instances.The subset selection is random.
  • C.4 EXTENDED TEMPORAL ANALYSIS: Across six temporal partitions grouped by issue-creation year, model % Resolved performance under oracle retrieval shows no consistent correlation with year.The analysis extends the temporal evaluation reported in Table 7 of the main paper and presents results in Table 21.

C.5 F2P, P2P RATE ANALYSIS · C.6 PATCH GENERATION EXTENDED ANALYSIS · C.7 SOFTWARE ENGINEERING METRICS

The extended analyses classify partially successful patches, compare generated edits with gold patches, and explore software-engineering metrics for evaluating patch complexity and code quality. They show that many applied patches make no progress, while metrics can reveal risks hidden by smaller edits.

  • C.5 F2P, P2P RATE ANALYSIS: C.5 defines six outcomes for successfully applied patches using whether F2P and P2P tests pass.The categories include Resolved, Breaking Resolved, Partially Resolved, Work in Progress, No-Op, and Regression.
  • C.5 F2P, P2P RATE ANALYSIS: 60% to 70% of non-Resolved No-Op or Regression cases are No-Op, with the remainder breaking existing behavior.These cases comprise the majority of non-Resolved applied patches and do not solve any F2P test case.
  • C.5 F2P, P2P RATE ANALYSIS: Partially successful categories are smaller overall, although manual inspection indicates some task understanding limited by missing inter-file dependencies.The passage identifies Breaking Resolved, Partially Resolved, and Work in Progress as the relevant categories and notes the potential value of execution feedback.
  • C.6 PATCH GENERATION EXTENDED ANALYSIS: C.6 finds model patch generations are generally closer in size to average gold edits when all oracle-setting patches are included.The analysis recalculates patch statistics regardless of whether patches applied successfully.
  • C.7 SOFTWARE ENGINEERING METRICS: Software-engineering metrics provide complexity and quality signals beyond semantic similarity by characterizing a patch’s changes and effects on the codebase.The exploratory evaluation uses Cyclomatic and Halstead measures, with Radon computing metrics directly from source code.
  • C.7 SOFTWARE ENGINEERING METRICS: The metric study applies successful Claude 2 Oracle patches for psf/requests and computes Cyclomatic and Halstead scores for modified functions.Each patch is applied to the codebase before calculating the metrics.
  • C.7 SOFTWARE ENGINEERING METRICS: A model patch can solve the problem with 6 instead of 11 lines and modify 1 instead of 2 files yet increase Cyclomatic complexity.In the case study, HTTPAdapter complexity rises from 3 →5 for the model patch, versus gold changes of 2 →3 and 0 →1.

D ADDITIONAL EXPERIMENTAL DETAILS … E SOCIETAL IMPACT

The additional details describe retrieval, inference, and prompting procedures for generating repository patches, while noting that prompt variations had limited effect. The paper also highlights safety concerns surrounding autonomous code generation, including faithfulness to human intent and guardrails against misinterpreted objectives.

  • D.1 RETRIEVAL DETAILS: Sparse retrieval prepends file paths to contents to improve retrieval when issue descriptions mention filenames directly.
  • D.1 RETRIEVAL DETAILS: Oracle retrieval extracts file paths from the reference patch while excluding test files.
  • D.2 INFERENCE SETTINGS: Each instance receives one generated patch, using greedy decoding for all models in a Pass@1-style evaluation.Generations are relatively expensive, motivating a single patch per instance.
  • D.3 PROMPT TEMPLATE EXAMPLE: The prompt supplies an issue statement together with a partial codebase containing README and source-file contents.The codebase is delimited with explicit file markers.
  • D.3 PROMPT TEMPLATE EXAMPLE: Models are instructed to return a single git-apply-compatible patch specifying changed files, line numbers, and removed or added lines.A patch can contain changes to multiple files.
  • D.3 PROMPT TEMPLATE EXAMPLE: The example patch demonstrates edits to code by showing unified-diff headers and changed function lines.The example includes modifications to Euclidean and Bresenham implementations.
  • D.3 PROMPT TEMPLATE EXAMPLE: Slightly changing the number of instruction or example lines did not substantially affect overall performance, aside from findings reported in Section 5.
  • E SOCIETAL IMPACT: Machine-automated software engineering raises AI-safety concerns about preserving human intent and adding guardrails when code agents misinterpret objectives.The paper frames these concerns as important ramifications of future automated software engineering.

F IN-DEPTH ANALYSIS OF SWE-LLAMA GENERATIONS

The qualitative analyses show that SWE-Llama can solve relatively localized issues but struggles with multi-line or multi-file changes, codebase understanding, and underspecified problems. Comparisons with gold patches reveal solutions that are correct yet less robust, overly simplistic, or insufficiently grounded in surrounding code.

  • Successful generations: SWE-Llama correctly solves some relatively short, localized issues when oracle retrieval supplies the relevant code context.Successful cases include reasoning from interactive Python behavior or error traces to identify semantically simple edits.
  • Failure modes: SWE-Llama struggles when fixes require coordinating changes across multiple lines or files and understanding broader codebase relationships.The analyses specifically identify difficulties with multi-line and multi-file changes, cyclic imports, and regex changes whose effects extend beyond the reported command.
  • Successful generations: The model’s correct patches can still use primitive implementations or overlook future compatibility and cleanup considerations present in the gold patch.Examples include reimplementing validation instead of using existing methods and accepting boolean-like values without removing related warnings.
  • Failure modes: Underspecified issues are especially challenging when models must map domain terminology to implementation logic without an explicit trace or precise localization.This occurs in tasks involving Django context variables and Sphinx title rendering, where grounding the issue language in the codebase is essential.
  • Failure modes: A further failure pattern is choosing a plausible but incorrect implementation despite reproducible evidence and a more appropriate existing helper.In one scikit-learn task, the gold patch uses svd_flip in a different location, whereas the model fails to resolve the parity inversion correctly.
Loading 2310.06770v3…