Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

1 Introduction

• Issue-resolution benchmarks focus on passing correctness tests, so they do not capture the runtime costs of different patches that fix the same issue.The study distinguishes repository-level issue resolution from prior generated-code performance studies at function or single-file level and repository optimization work.Source: §1 · pp. 1–2
• The study measures instruction counts and wall-clock times by repeatedly running built-in F2P tests, using Welch ANOVA to compare correct patches.It examines patches from four frontier models paired with SWE-agent and developer ground-truth patches on verified SWE-bench-family splits.Source: §1 · pp. 2–3
• 41.98% of cases show significant instruction-count differences and 34.73% show significant wall-clock-time differences among solutions under Welch ANOVA.The study also reports that solution diversity is widespread and that the model with the highest resolution rate produces only the third-most efficient patches.Source: §1 · p. 2
• Issue-resolution rates and patch runtime are essentially uncorrelated for the compared models, with nearly reversed rankings across the measures.The model leading in resolution rate produces only the third-most efficient patches, and developer ground-truth patches have no systematic performance advantage over model-generated patches.Source: §1 · p. 2
• Performance comparisons expose architectural regressions that correctness evaluation misses, including costly fixes that silently reintroduce a heavy dependency the developer removed.The three repair strategies in the case study differ in performance by more than a factor of two.Source: §1 · p. 3

2.1 Datasets and Task

• The benchmarks pair each issue with a base commit, developer patch, F2P tests, and regression tests; resolution requires passing F2P tests while keeping P2P tests passing.Because correctness does not require reproducing the reference implementation, distinct fixes can pass the same tests.Source: §2.1 · pp. 3–4
• Repeated F2P execution is used as a performance workload because it exercises behavior changed by the fix; P2P tests are excluded from measurement.Preparation and assertion costs are included, which can dilute measured differences, while repeated runs average execution noise.Source: §2.1 · p. 4

2.2 Filtering for Measurability

• The corpus requires one F2P test per instance, repeatable execution within one process, and no input/output operations in the test.The one-test rule avoids merging measurements; dynamic probing checks repeatability and I/O behavior.Source: §2.2 · p. 4, §2.2.1 · p. 4, §2.2.2 · pp. 4–5

2.2.2 Dynamic Probing: Repeatability and I/O Freedom.

• Decorator-based probing runs the F2P test as a complete unit, excluding tests that fail under repetition or perform I/O.The method avoids modifying test logic and excludes I/O because it can drown out CPU-level performance signals.Source: §2.2.2 · p. 5, Figure 1 · p. 5

2.2.3 Filtering Result.

• Filtering raw datasets with static rules and dynamic probing yields 339 instances; repeatability removes 87, and the I/O-free rule removes 206, including tests whose behavior or scaffolding performs I/O.Source: §2.2.3 · pp. 5–6

2.3 Model Solving and Patch Diversity

• The four models are paired with SWE-agent; Claude has the highest resolution rate on both datasets, while DeepSeek has the lowest.The overall correctness ordering is Claude ≫ Qwen > GPT ≈ DeepSeek.Source: §2.3 · pp. 6–7
• 262 cases (77.29%) contain at least two distinct correct solutions, forming the study corpus for comparing patch performance.In 11.21% of cases, all four models produce patches distinct from one another and from the ground truth.Source: §2.3 · p. 8, Table 3 · p. 8

3.1 Execution Protocol

• Each distinct patch is measured by repeatedly running the native F2P test inside its official per-instance Docker environment.The inner loop repeats the test within one process, while five fresh-process outer-loop replicates capture run-to-run variation.Source: §3.1 · p. 8, §3.1.2 · p. 9

3.1.1 Inner loop.

• A calibrated inner-loop count targets 4–8 seconds for the fastest solution, averaging 6.8 × 10^5 invocations per case and a median of 1.6 × 10^4.Calibration increases repetitions until the fastest solution’s measurement falls within the target window.Source: §3.1.1 · p. 9

3.2.1 Dual Metrics.

• Instruction count measures computation through hardware counters, while wall-clock time captures end-to-end latency and environmental effects.The metrics are complementary: instruction count is less sensitive to core contention and frequency scaling, while wall-clock time is more prone to fluctuations.Source: §3.2.1 · p. 9

3.2.2 Discrimination assessment.

• Welch ANOVA compares solution performance while accounting for both within-solution variation and unequal variances, unlike RSD’s reliance on replicate means.The analysis log-transforms measurements so differences correspond to ratios on the original scale, and uses p < 0.05 to identify statistically distinguishable performance.Source: §3.2.2 · pp. 9–10
• The analysis transforms measurements logarithmically before testing, making differences on that scale correspond to performance ratios on the original scale.Welch’s test statistic and p-value account for within- and between-solution variation; the paper applies p < 0.05 as its significance threshold.Source: §3.2.2 · p. 10

3.3.1 Core reservation.

• Measurements use shared server infrastructure, with two physical cores reserved for Docker management and one for the operating system.The reservations confine background container and image activity away from measurement cores.Source: §3.3 · p. 10, §3.3.1 · p. 11

3.3.2 Kernel-level CPU isolation.

• Disjoint cgroup cpuset partitions isolate groups of four physical cores, with each measured case using one logical core per physical core.The sibling logical cores remain idle, reducing shared-core cache contention; up to seven cases can run in parallel.Source: §3.3.2 · p. 11

3.3.3 Memory limits and NUMA binding.

• Containers are bound to the server’s single NUMA node, capped at 4 GB of memory, and run with swap disabled.These constraints are intended to keep memory local and predictable and to avoid paging-related disk traffic.Source: §3.3.3 · p. 11

4.1 Test Discriminative Property

• Nearly half of cases show statistically distinguishable patch performance, with instruction count more discriminative than wall-clock time.The study reports similar shares of distinguishable cases on SWE-bench and SWE-bench-Live.Source: §4.1 · pp. 12–13
• Welch ANOVA identifies some differences at low RSD, while RSD can also be elevated by run-to-run noise rather than genuine patch differences.In one case, Welch ANOVA found a difference at 0.075% instruction-count RSD; another high-RSD case reflected test randomness.Source: §4.1 · pp. 12–13

4.2.1 Speedup Comparisons.

• The study aggregates per-case speedup ratios with a geometric mean and reports results for both the full corpus and distinguishable subsets.The ground-truth patch is treated as one solution among others, so pairwise comparison orientations are interchangeable.Source: §4.2.1 · pp. 13–14
• GPT and DeepSeek lead on runtime performance, followed by Claude and then Qwen, reversing the resolution-rate ordering.The same model ordering holds on the full and distinguishable subsets.Source: §4.2.1 · p. 15, Table 7 · p. 15

4.2.2 Champion Distribution.

• DeepSeek produces the most fastest solutions, with GPT next, while Qwen and Claude rank lower; the gaps widen on distinguishable subsets.The champion distribution splits credit evenly when multiple sources produce the same fastest solution.Source: §4.2.2 · pp. 15–16, Figure 4 · p. 16

4.3 Case Study

• In astropy__astropy-7671, the patches use different parser strategies and vary in performance; some reintroduce the heavy dependency that correctness tests miss.The ground truth and DeepSeek retain the lightweight LooseVersion parser, while GPT and Qwen use pkg_resources unconditionally and Claude falls back to it.Source: §4.3 · pp. 17–18, Figure 5 · p. 17

Internal validity

• The method assumes data-preparation and assertion costs are equal across correct solutions, although some patches may alter those costs and contaminate measurements.The method cannot detect this contamination.Source: §5 · p. 18
• Repeated F2P runs may execute repair-relevant code only in some invocations, reducing discrimination when later runs merely read retained state.The measurements may then degenerate to indistinguishably small values.Source: §5 · p. 18

External validity

• External validity is bounded by a deterministically filtered measurable corpus, Python-only benchmarks, and solutions generated with one SWE-agent framework.The authors plan to extend the work to C and C++ and include more frameworks.Source: §5 · p. 18

Code performance optimization

• Repository-level optimization work targets already-correct code, whereas this study examines patches produced to resolve functional issues.Prior work includes optimization diffs for C/C++ [7], repository-level optimization [19] [28] [37], and related methods [11] [31] [35].Source: §6 · p. 18

Efficiency-aware code generation

• Efficiency-aware generation and benchmarks evaluate generated code mainly at the function or single-file level, rather than in real-world repositories.This line includes efficiency-aware training and prompting, runtime-feedback methods, and instruction-count ranking [5] [12] [14] [20] [21] [22] [32] [33] [34] [45].Source: §6 · p. 18

Empirical studies of AI-generated code efficiency

• Empirical studies measure AI-generated code efficiency across benchmarks and tools, while this paper focuses on patches for repository issues.Prior studies include runtime regressions in AI coding tools and code-efficiency benchmarks [2] [6] [9] [27] [30].Source: §6 · pp. 18–19

• At most 42% of cases show statistically distinguishable differences among correct patches under Welch ANOVA.The analysis measures instruction counts and wall-clock times using built-in F2P tests on verified SWE-bench and SWE-bench-Live splits.Source: §7 · p. 19
• The measurable corpus contained 339 instances, including 262 cases with at least two genuinely different patches from four frontier models and the developer.Source: §7 · p. 19
• Issue resolution rate and runtime performance are independent model-capability dimensions, with nearly reversed orderings.The authors call for future work to consider generated-patch performance alongside issue resolution rate.Source: §7 · p. 19

• The injection finds the F2P function definition by matching its def statement and inserts @measure_all before it.Source: A Decorator Implementation for Filtering · p. 21
• PF_TRACE_START and PF_TRACE_END markers bracket the measured pass to delimit the traced window.The wrapper emits the start marker before calling the function and the end marker after it returns.Source: A Decorator Implementation for Filtering · pp. 21–22

• The repeatability exclusions comprise 9 cases that cannot execute, 12 that cannot be isolated, and 66 that cannot be repeated.Execution failures include unavailable dependencies or pathological environments; isolation failures include corrupted F2P identifiers or tests dependent on prior state; repeatability failures arise from persistent state changes or network hangs.Source: B Excluded Instances by In-process… · p. 22, Table 9 · p. 22
• The RSD density figure groups cases by distinct-solution count k, with instruction count on the left and wall-clock time on the right.Source: Figure 6 · p. 23

• Each replicate measurement is log-transformed, and each solution’s weight is inversely proportional to the variance of its mean.The analysis uses the replicate mean and sample variance on the log scale to form the weight.Source: C Welch ANOVA Details · p. 23
• The Welch statistic compares weighted between-solution spread against a correction term that absorbs variance heterogeneity.Source: C Welch ANOVA Details · p. 23
• Under the null hypothesis of equally fast solutions, the Welch statistic approximately follows an F distribution, from which the p-value is obtained.The distribution uses k−1 and the Satterthwaite approximate denominator degrees of freedom.Source: C Welch ANOVA Details · p. 23

D RSD Distribution by Number of Distinct Solutions

• As the number of distinct solutions k grows, the between-solution RSD distribution shifts right.Source: D RSD Distribution by Number of… · p. 23
• As k grows, the distribution peak flattens and its tail extends further.Source: D RSD Distribution by Number of… · p. 23
• The same RSD value falls at a different quantile in each group, making one threshold for all k infeasible.Source: D RSD Distribution by Number of… · p. 24

E Within-solution RSD and Case Study

• Wall-time within-solution noise is 1–3% and two to three times instruction-count noise; one case reaches 36.80% and 36.41%, respectively.The 36.80% instruction-count and 36.41% wall-time maxima occur in sympy__sympy-20428; Table 10 summarizes within-solution RSD across cases.Source: E Within-solution RSD and Case Study · p. 24, Table 10 · p. 24
• In sympy__sympy-20428, per-process randomness creates two execution-cost modes about 2× apart, producing 36% within-solution RSD.Fresh-process replicates take fast or slow paths through SymPy’s assumption engine, which falls back to minimal-polynomial computation and integer polynomial factorization.Source: E Within-solution RSD and Case Study · p. 24
• The case’s two patches share a one-line symbolic zero-test fix, but their 10.8% between-solution RSD reflects luck across five replicates rather than a clear solution difference.The model patch took the slow path in 1 of 5 replicates and ground truth in 2 of 5.Source: E Within-solution RSD and Case Study · p. 24
• Welch ANOVA weighs between-solution spread against within-solution noise and correctly classifies the case as indistinguishable, unlike RSD alone.Source: E Within-solution RSD and Case Study · p. 24