Source-linked AI summary
SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, Zejun Ma
TL;DR
Repository-level code performance optimization remains largely unexplored despite its importance in real-world software engineering. SWE-Perf benchmarks the task using performance-improving pull requests and evaluates models in file-level and repo-level settings; the evaluations reveal substantial gaps between existing models and expert-level optimization.
Problem
LLM performance optimization at repository level remains largely unexplored, and existing benchmarks do not capture the cross-file and module complexity of real-world optimization.
Source: §1 · p. 1Method
SWE-Perf is a repository-level benchmark built from performance-improving pull requests, with codebases, target functions, performance tests, expert patches, and executable environments.
Source: §1 · pp. 1–2Results
Existing models show substantial gaps from expert-level optimization; OpenHands performs best, yet an 8.59% performance gap from Expert persists.
Source: §5.2 · p. 8, §6 · p. 9Takeaways & Limitations
SWE-Perf provides a benchmark and baseline assessments that expose the complexity of repository-scale optimization and the room for improvement in current models.
Source: §6 · p. 9The benchmark supplies target functions; whole-codebase optimization without those targets is left for future work.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.
1. Introduction
Repository-level performance optimization remains underexplored compared with code correctness, despite its importance in real-world software engineering. SWE-Perf benchmarks this task using real repositories, performance tests, and expert patches, and evaluations reveal a substantial gap between current models and experts.
1. Introduction
- Existing repository-level benchmarks emphasize code correctness, while performance benchmarks largely target isolated functions rather than repository-scale optimization.This misses cross-file and cross-module optimization challenges that characterize real-world practice.Source: §1 · p. 1
- SWE-Perf evaluates LLMs on performance optimization in real-world repositories by applying generated patches and measuring the modified codebase.The workflow supplies source code and performance-related tests, using expert performance as a reference.Source: §1 · p. 1, Figure 1 · p. 2
- The benchmark contains 140 examples from 9 popular GitHub repositories, selected by testing performance across 100K pull requests.Each example includes a codebase, target functions, performance tests, an expert patch, an executable environment, and runtime metrics; the expert patch is a human-derived gold standard.Source: §1 · p. 2
- Evaluations in file-level oracle and repo-level realistic settings show substantial gaps between all evaluated LLMs and expert performance.The results highlight unresolved challenges in meeting practical requirements for performance-aware code optimization.Source: §1 · p. 2
- The paper contributes a repository-level benchmark, a data-collection pipeline, and performance-specific evaluation metrics applied to several LLMs.The pipeline and metrics are intended to support systematic evaluation of repository-level performance optimization.Source: §1 · p. 2
2. Related Work
Prior code-efficiency benchmarks mainly evaluate function-level optimization, while repository-level software engineering benchmarks focus on tasks with defined objectives such as bug fixing. Existing SWE approaches likewise were not designed for open-ended performance optimization.
Code Efficiency
- Function-level efficiency benchmarks overlook cross-file and cross-module challenges, including dataflow refactoring and parallelism, where optimization potential is typically greater.The cited benchmarks include Mercury, EFFIBENCH, EvalPerf, PIE, and KernelBench.Source: §2 · p. 2
Repository-Level SWE Tasks
- Repository-level SWE datasets primarily address well-defined tasks such as bug fixing, whereas efficiency optimization is open-ended and requires target identification, performance-oriented changes, long-context understanding, planning, and domain knowledge.GSO identifies performance-improving commits using an LLM judge and code-change heuristics; this work uses pull requests and runtime environments.Source: §2 · p. 2
Approaches to SWE Tasks
- Pipeline-based and agent-based approaches support repository-level SWE tasks, but were not designed for open-ended performance optimization.Examples include Agentless and SWE-Fixer for pipelines, and OpenHands, SWE-Agent, SWE-Smith, and SWE-Search for agent-based systems.Source: §2 · p. 3
3. SWE-Perf Dataset
SWE-Perf defines a repository performance-optimization task and constructs its dataset from pull requests whose runtime gains can be measured and verified. The collection process screens for correctness and performance improvement, attributes gains to patched code, and verifies stable results before extracting optimization targets.
3. SWE-Perf Dataset
- SWE-Perf is a dataset built from popular GitHub repositories to evaluate LLM performance optimization on real-world software.The dataset section describes task formulation, collection, and data statistics and distribution.Source: §3 · p. 3
3.1. Task Formulation
- The task takes a codebase and target functions as input and produces a performance-optimization patch or code that can be applied to create a new codebase.Target functions restrict evaluation to performance-related tests; this guidance addresses the difficulty and cost of identifying relevant regions across large repositories.Source: §3.1 · p. 3
- Target functions constrain the task scope; the authors encourage future work to explore whole-codebase optimization without them as models improve.This avoids requiring models and evaluation to identify relevant regions without guidance.Source: §3.1 · p. 3
3.2. Data Collection
- Collection begins with popular repositories and issue-resolving pull requests, then measures unit-test runtimes for original and modified codebases to find performance improvements.The process builds executable Docker environments, runs tests with pytest, and records per-test runtimes; failed builds or runtime collection are excluded.Source: §3.2 · p. 3
Phase 3: Identify PRs with Performance Optimizing Pull
- Performance screening keeps passing tests with substantial runtime improvement and checks that their execution covers patched code without running tests modified by the pull request.Runtime measurements use repeated runs; outliers are filtered using the interquartile-range method.Source: Phase 3: Identify PRs with Performance… · p. 4
3.2. Data Collection
- The pipeline verifies stable performance gains after warm-up and 20 test repetitions before extracting optimization targets for Oracle and Realistic settings.The figure identifies stable-improvement verification and target extraction as later pipeline stages.Source: Phase 3: Identify PRs with Performance… · p. 4, Figure 2 · p. 4
4. Calculate statistical performance. To confirm stable per-
SWE-Perf establishes stable performance gains using statistically tested runtime comparisons, then packages repository-level optimization tasks with both file-level and repository-level targets. The final dataset contains 140 instances drawn from nine widely used GitHub repositories.
4. Calculate statistical performance. To confirm stable per-
- A test case enters the final SWE-Perf dataset when its statistically significant minimum performance gain δ, computed by conservatively adjusting the modified runtime distribution against the original, exceeds 0.05.Significance is assessed with a Mann–Whitney U test at threshold 0.1.Source: §4 · p. 5
- Oracle tasks target functions directly modified in the expert patch, while Realistic tasks target measured functions that may benefit from repository-wide changes.Oracle provides related files as context; Realistic permits repository-wide edits and requires additional capabilities such as retrieval.Source: §4 · p. 5
- Target functions are identified from functions dynamically executed by performance tests, avoiding the tests themselves to prevent functional-pruning leakage.The pipeline combines dynamic execution traces from yappi with AST parsing.Source: §4 · p. 5
- The dataset includes original codebases, executable environments, target functions, performance tests, runtime metrics, and expert patches.Expert patches provide human-level performance-optimization references.Source: §4 · pp. 5–6
3.3. Data Statistics and Distribution
- 140 instances were retained from 102,241 collected pull requests, with the final examples drawn from nine repositories.The filtering pipeline successively produced 19,797 PRs, 1,696 valid instances, and then the final 140.Source: Figure 3 · p. 5, §3.3 · p. 6
4. Evaluation Methodology
The evaluation applies generated patches, checks functional correctness, and measures performance on original and modified codebases. It uses runtime controls and progressively stricter metrics to distinguish patch applicability, correctness, and performance gain.
4. Evaluation Methodology
- The evaluation applies each generated patch and runs performance-related tests on original and modified codebases, recording test results and runtimes.Original runtimes are remeasured during testing to make original and post-patch measurements comparable despite environmental variability.Source: §4 · p. 6
- Figures 5 and 6 report performance for correct examples, with expert performance calculated using the corresponding method’s correct examples.Source: Figure 5 · p. 7, Figure 6 · p. 7
- Apply measures successful patch application, while Correctness requires every unit test to pass after the patch.The correctness metric uses the logical AND of the post-patch test results.Source: §4 · p. 7
- Performance measures statistically significant minimum runtime gain after warm-up, 20 repetitions, and outlier filtering.The per-test minimum gain is calculated before the benchmark performance score is computed.Source: §4 · p. 7
5. Experiments
The experiments compare file-level oracle prompting and repository-level baselines, finding OpenHands strongest but still below experts overall. Further analyses examine correctness, runtime, target-function count, and patch keywords.
5.1. Baselines
- The baselines include oracle file-level prompting, Agentless’s staged pipeline, and OpenHands’s iterative agent; Agentless and OpenHands use Claude-3.7-sonnet.Oracle prompting uses expert-patch files and target functions for single-pass patch generation.Source: §5.1 · p. 7
5.2. Main Results
- OpenHands performs best, but an 8.59% performance gap remains between OpenHands and Expert.OpenHands outperforms the Expert by 0.4% on sklearn.Source: §5.2 · p. 8
5.3.1. PERFORMANCE ANALYSIS DECOUPLED FROM CORRECTNESS
- When metrics count only correct examples, OpenHands achieves approximately 3% higher benchmark performance than alternative methods using identical models.The Expert metric is calculated using only correct samples from the corresponding method.Source: §5.3.1 · p. 8
5.3.2. IMPACT OF RUNTIME ON PERFORMANCE
- With longer runtimes, expert performance rises while model performance plateaus, leaving model improvement on long-runtime cases needing further work.The analysis links longer runtime with a higher expert performance ceiling.Source: §5.3.2 · p. 8
5.3.3. IMPACT OF TARGET FUNCTIONS ON PERFORMANCE
- As target-function counts increase, expert performance declines and the OpenHands–Expert gap widens, making multi-function optimization increasingly difficult.At lower function counts, OpenHands performs comparably to the Expert.Source: §5.3.3 · pp. 8–9
5.3.4. KEYWORD ANALYSIS FOR PERFORMANCE
- OpenHands patches emphasize low-level structures and infrastructure, including children, identifiers, and environment or module handling.The word-cloud analysis also notes terms such as time and attributes.Source: §5.3.4 · p. 9, Figure 10 · p. 9
- Expert patches emphasize high-level abstractions, data integrity, and domain-specific performance work, including type-related terms, sympy, and workspace.The analysis associates these terms with type safety, data-value handling, and core computational workflows.Source: §5.3.4 · p. 9, Figure 11 · p. 9
6. Conclusion
SWE-Perf introduces a repository-level benchmark for realistic code performance optimization, an area largely unexplored in prior LLM evaluations. Its baseline assessments show substantial gaps between current models and expert-level optimization.
6. Conclusion
- SWE-Perf provides the first repository-level dataset focused on realistic code performance optimization, a task traditionally reliant on human expertise and largely unexplored in prior LLM evaluations.The conclusion calls for approaches that handle cross-module and repository-scale optimization in real-world software.Source: §6 · p. 9
Impact Statement
The dataset uses publicly accessible open-source repositories whose licenses permit research use, selected using popularity metrics. The benchmark and its tools and results are released publicly.
Impact Statement
- SWE-Perf selects research-permitted open-source repositories using popularity metrics such as stars and forks, and releases its instances, tools, and results publicly.The stated selection aim is to avoid intentional or unintentional bias toward project types, communities, or contributor demographics.Source: Impact Statement · pp. 9–10
A. Limitations
The authors identify limited repository coverage and the use of human-written patches as ground truth as limitations of SWE-Perf.
A. Limitations
- The limited open-source repository set constrains coverage and generalizability, while human-written ground-truth patches may underestimate the true upper bound of improvement.The authors suggest expanding the dataset and note that human patches may not be optimally performant.Source: A. Limitations · p. 14
B. Dataset
The section describes the experimental hardware and measurement thresholds chosen to balance statistical reliability with benchmark-construction cost.
B. Dataset
- The experiments ran on two Linux machines, each with 256 logical CPU cores and 2.0 TiB of RAM.Each machine had 128 physical cores across two sockets, with two threads per core.Source: B. Dataset · p. 14
B.1. Threshold Choices
- 30% runtime improvement was required in Phase 3, corresponding to roughly 2.7 times the observed measurement noise.The average runtime coefficient of variation was 10.976%.Source: §B.1 · p. 14
- 20 repeated measurements and α = 0.1 were used for Phase 4 testing, with δ > 0.05 required for a practically meaningful gain.These choices balance an effective Mann–Whitney U test sample size with computational feasibility, while filtering gains that are detectable but too small.Source: §B.1 · p. 14
B.2. Details of Data Statistics
The appendix summarizes dataset statistics, describes the benchmark’s optimization-strategy coverage and scope, and documents baseline settings and supplementary prompt materials.
B.2. Details of Data Statistics
- Tables summarize phase-by-phase data volume, successful pytest runtimes in minutes, and repository abbreviation mappings.Runtime statistics cover Phase 2, Step 2; test runtime is per-codebase pytest time, while total duration also includes operations such as Docker image building.Source: Table 3 · p. 14, §B.2 · p. 14, Table 4 · p. 15
C.2. Additional Word Clouds
- Supplementary materials show added-line word clouds for Agentless and Oracle patches and provide the Oracle repository-optimization prompt.The prompt asks for repository-wide efficiency and speed improvements while preserving existing unit tests.Source: Figure 12 · p. 16, Figure 13 · p. 16, §C.2 · pp. 16–17
B.3. Coverage and Representativeness
- Optimization strategies span four patterns, led by Cache & Compatibility (32.9%), while coverage remains concentrated in Python repositories.The categories are Redundancy Elimination (25.7%), Algorithmic Logic (26.4%), Data & Resource Optimization (15.0%), and Cache & Compatibility (32.9%).Source: §B.3 · pp. 14, 16
C.1. Details of Baselines
- Baselines include Oracle prompting, Agentless with one sample, and OpenHands with up to 2,000 iterations.The listed model versions span OpenAI/GPT, Claude, DeepSeek, Gemini, and Qwen.Source: §C.1 · p. 16