PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, Parthasarathy Ranganathan

arXiv:2511.06090v3cs.SEcs.AIcs.PF

TL;DR

Most software-engineering benchmarks focus on what code gets fixed, while repository-level performance optimization on real workloads remains insufficiently evaluated. SWE-FFICIENCY builds a benchmark from real repository tasks and evaluates agents against expert speedups while checking correctness. Agents average less than 0.23× expert speedup and often introduce correctness bugs.

  • Problem

    Most language-model software-engineering benchmarks assess what gets fixed rather than implementation properties such as runtime, and existing repository-optimization benchmarks do not fully test investigation of code semantics and correctness tests.

    Source: §1 · pp. 1–2

  • Method

    SWE-FFICIENCY builds 498 real-repository optimization tasks using a pipeline that combines filtering, coverage-based test selection, manual workload annotation, and execution validation.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Agents achieve less than 0.23× expert speedup on average and often introduce correctness bugs, with difficulties localizing expert opportunities and favoring superficial speedups.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark and its released artifacts provide a basis for research on automated performance engineering and long-horizon software reasoning.

    Source: §1 · p. 2, §6 · p. 8

    The benchmark primarily covers Python and Cython changes in nine libraries; extending it to lower-level stacks requires generalizing test selection and language-specific build awareness.

    Source: §6 · p. 8

Abstract

from arXiv · show

Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.23x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.

1. Introduction

Most software-engineering benchmarks assess what language models fix rather than implementation properties such as runtime, while existing repository-optimization benchmarks do not fully test the investigation of code semantics and relevant correctness tests. SWE-FFICIENCY evaluates that workflow on real workloads and repositories, and its evaluation finds agents far behind expert speedups, with correctness bugs and difficulties localizing expert opportunities.

1. Introduction

  • Most benchmarks emphasize what gets fixed rather than code properties such as runtime, and recent performance benchmarks avoid real-world, end-to-end repository workloads.Software optimizations can have substantial impact; cited work reports pure software changes reducing high-utilization datacenter workload throughput by 10%.Source: §1 · p. 1
  • Existing repository-optimization benchmarks use oracle scripts or performance-adapted tests, but do not fully assess recovering code semantics and correctness from an unfamiliar codebase.Performance engineers must characterize a workload, localize an intervention, and identify existing unit tests that can check for regressions.Source: §1 · pp. 1–2
  • SWE-FFICIENCY evaluates language-model optimization of real-world workloads through repository edits, using 498 tasks across nine data-science, machine-learning, and HPC codebases.Its pipeline combines attribute filtering, static analysis, code coverage, and execution validation; speedup ratio compares model edits with expert edits.Source: §1 · p. 2
  • Agents achieve less than 0.23× expert speedup on average and often introduce correctness bugs, while struggling to localize expert opportunities and preferring superficial speedups.The authors identify repository-level reasoning, systems optimization, and long-horizon planning as areas needing advances to close the expert gap.Source: §1 · p. 2
  • The benchmark, evaluation harness, model trajectories, dataset, and pipeline are released publicly to support reproduction and research on automated performance engineering.The paper frames targeted workload runtime optimization as a foundational capability before more complex multi-objective regression management and production deployment.Source: §1 · p. 2

2. SWE-FFICIENCY Overview

SWE-FFICIENCY is a benchmark of real performance-optimization pull requests, built through a multi-stage process that selects reproducible tasks with expert patches, relevant correctness tests, and annotated performance workloads. It lets agents choose their optimization approach while evaluating speedup against expert edits and correctness with repository tests.

2. SWE-FFICIENCY Overview

  • SWE-FFICIENCY turns real performance-optimization pull requests into tasks requiring a faster workload and preservation of existing repository-test correctness.Its benchmark design distinguishes performance workloads from correctness tests, reflecting that repositories generally keep unit tests fast and separate from performance benchmarks.Source: §2 · p. 2, §2.2 · p. 4

2.1. Data Collection Procedure

  • The collection pipeline scrapes popular Python repositories and filters pull requests, selects tests covering edited code, manually annotates workloads, and validates reproducible speedups.Workload scripts are manually written using pull-request descriptions and discussion because ad-hoc demos are not reliably extractable and tested language-model generation does not consistently elicit claimed gains.Source: §2.1 · pp. 2–4

2.2. SWE-fficiency Dataset Distribution and Unique Benchmark Features

  • Agents may use any edit strategy, with correctness grounded in unit tests and performance measured against the expert patch’s speedup.The benchmark targets pass-to-pass optimization: speeding up already-correct code without introducing new behavior.Source: §2.2 · p. 4

2.3. Task Formulation

  • Agents receive a complete codebase and performance workload, and a task’s speedup ratio is the LM patch’s speedup divided by the gold patch’s speedup.A patch that fails unit tests or is empty receives a clamped score; aggregate speedup ratio uses a harmonic mean with a per-instance lower bound.Source: §2.3 · pp. 4–5

Why a factor-based evaluation metric (not % solved)?

  • Speedup ratio supports continued comparison after systems surpass experts, unlike percentage-style metrics that converge toward saturation.The paper states that the metric motivates progress both when systems score around 0.23× of expert performance and if they later exceed 1×.Source: Why a factor-based evaluation metric… · p. 5

3. Evaluation Setup

The evaluation uses containerized tasks on fixed hardware, with CPU and memory resources pinned for workers, and measures 15 frontier models using one trajectory per instance. The setup is designed for reproducibility and pass@1 evaluation.

Machine and Evaluation Configuration

  • Each task runs in a Docker image on a single 64-vCPU, 256GB VM, with workers pinned to four physical CPU cores and limited to 16GB memory.The authors report less than 0.5% speedup-ratio variance across independent runs on the fixed hardware platform.Source: §3 · p. 5

Agent Scaffold

  • OpenHands and SWE-Agent provide file-editing and terminal tools, with a three-hour task limit and up to 100 actions per task.The SWE-Agent setup imposes a $1 token-spend maximum for its evaluated models, whereas OpenHands has no dollar-cost limit.Source: §3 · p. 5

Models

  • 15 frontier models from six organizations are evaluated with one sampled trajectory per instance, using pass@1 to match single-submission expert PRs and realistic agent workflows.Agents can still iterate over edits and rerun workloads within that trajectory.Source: §3 · p. 5

4. Experiments and Results

Leading agents achieve less than 0.23× expert speedup on average and often introduce correctness bugs; the analyses identify difficulty, mislocalization, and shortcut-oriented edits as recurring gaps.

4.1. Overall Performance

  • On average, agents achieve less than 0.23× expert speedup and often introduce correctness bugs.The broader evaluation also finds substantial gaps in localization and principled optimization.Source: §4 · p. 5, §4.1 · p. 5
  • For all but GPT-5, CLAUDE 4.1 OPUS, and CLAUDE 4.5 SONNET, fewer than a quarter of solutions are both correct and faster than expert patches.Even functionally correct patches are often slower than the expert.Source: §4.1 · pp. 5–6
  • Agents match experts on lower-complexity tasks but struggle with longer workloads and larger expert speedup opportunities; performance changes little with expert patch size.The authors suggest the central difficulty is choosing where to intervene rather than how large an edit to make.Source: §4.1 · p. 6
  • Over 68% of expert gains occur in functions the LM never edits, although agents and experts modify the same files more than 55% of the time.Agents can choose a different function and miss gains available at a shallower scope.Source: §4.1 · p. 6

4.2. Qualitative Analysis

  • When agents underperform experts, they often stop after finding a measurable speedup rather than continuing to search for expert parity.Underperforming trajectories have median action counts below the 100-action cap, typically 30–50 turns.Source: §4.2 · p. 6
  • LMs favor localized shortcuts over structural cost reduction, while experts more often use systemic changes and localized, composable edits.Experts may use faster compiled backends or avoid slow object-dtype materialization; their edits are described as more maintainable.Source: §4.2 · pp. 6–7
  • Some LM patches hard-code workload-specific shapes, dtypes, or index structures, so their speedups may not survive minor input variations and can break unrelated callers.The paper reports examples including returning an input DataFrame unchanged and module-level monkey-patching.Source: §4.2 · p. 7
  • Manually annotated workloads show stronger performance deltas than LM-generated workloads in 76% of comparisons; 47% of LM workloads show no significant speedups.The comparison measures each expert patch under the two workload-generation approaches.Source: §4.2 · p. 7

5. Related Work

Prior work studies code efficiency at the function level and repository-scale bug-fixing, while repo-level performance benchmarks leave key localization skills insufficiently assessed.

5. Related Work

  • Function-level efficiency benchmarks assess model-generated functions or user-level workloads while holding utilized libraries fixed, unlike SWE-fficiency’s library-maintainer setting.Examples include MERCURY (Du et al., 2024), EFFIBENCH (Huang et al., 2024), PIE (Shypula et al., 2024), ECCO (Waghjale et al., 2024), KERNELBENCH, and ALGOTUNE (Press et al., 2025).Source: §5 · p. 7
  • Repository-scale benchmarks such as SWE-BENCH, COMMIT0 (Zhao et al., 2024), and SWT-BENCH (Mündler et al., 2024) mainly target bugs, features, and tests.Agentic systems such as SWE-agent (Yang et al., 2024) and OpenHands (Wang et al., 2025) provide tools for navigating and editing repositories.Source: §5 · p. 7
  • GSO and SWE-PERF evaluate repository-level runtime, but their designs insufficiently assess workload, bottleneck, edit, and in-repository test localization.SWE-PERF instructs agents to optimize specified functions; GSO supplies correctness tests and performance workloads without exposing those workloads to agents.Source: §5 · pp. 7–8

6. Discussion

SWE-FFICIENCY evaluates repository-level performance optimization through tasks pairing workloads, expert patches, and correctness tests. Current agents remain below expert performance, while the benchmark’s scope and workload settings leave room for expansion.

6. Discussion

  • The benchmark primarily covers Python and Cython changes in nine libraries; lower-level stacks, longer workloads, heterogeneous non-CPU hardware, and multi-objective metrics remain extensions.Extending to C, C++, or Rust would require generalized test selection and language-specific build awareness.Source: §6 · p. 8
  • Current agents remain well below expert performance, with mislocalization and shortcut bias among the observed gaps.The benchmark motivates progress toward autonomous performance engineering and software-engineering agents.Source: §6 · p. 8
  • 498 tasks pair performance workloads and expert patches with correctness tests covering the expert diff, enabling pass-to-pass optimization evaluation.The pipeline combines correctness checks, static analysis, coverage-guided test selection, manual workload annotation, and reproducibility checks.Source: §6 · p. 8

Impact Statement

The paper presents repository-level performance optimization as potentially reducing compute and energy use, while noting risks from autonomous code modification. It releases benchmark materials and reports that language models assisted with writing and compliance checks.

Impact Statement

  • Automated performance engineering could reduce the compute and energy footprint of software used in data science and machine learning.It could also lower the expertise barrier to optimization in open-source ecosystems.Source: Impact Statement · p. 9
  • Autonomous code-modification agents can introduce correctness regressions, maintainability debt, and workload-specific shortcuts that fail to generalize.The authors encourage retaining evaluation guardrails when deploying generated optimizations in production libraries.Source: Impact Statement · p. 9
  • The authors will release task instances, evaluation infrastructure, and model trajectories with documentation and community channels.The release is intended to support continued improvement of SWE-FFICIENCY.Source: Impact Statement · p. 9

A. LLM Usage

  • Language models helped polish the writing, correct grammar and typos, and check compliance with ICML’s author guide.The authors state that models were not used elsewhere in writing the work, beyond benchmark evaluations and experiments.Source: A. LLM Usage · p. 12

B. Additional Dataset Information

The appendices describe the benchmark’s task distribution and workloads, a taxonomy for expert patch mechanisms, and measurement choices. They also explain why workload-specific optimization reflects accepted production practice.

B. Additional Dataset Information

B.1. Repository Descriptions and Permissive Licenses

  • All task instances come from public repositories with permissive licenses, with repository-specific licenses listed in the appendix.The repositories are identified in Table 4.Source: §B.1 · p. 12, Table 4 · p. 13

B.2. Example Performance Workloads

  • Workloads report mean and standard deviation across repeated runs, and curation rejects instances without statistically significant execution improvements.Each workload may include setup separate from the measured repository functionality.Source: §B.2 · p. 12

B.3. LM Classification of Gold Patch Edit Types

  • Gold patches span categories such as algorithmic or data-structure improvements, memory efficiency, concurrency, I/O, code simplification, compiler tuning, configuration, and caching.The categories were generated with an LM and reviewed through a random sample of 50 classifications; agents remain free to use any optimization.Source: §B.3 · pp. 14–15

B.4. Dataset Schema

  • The dataset schema and its column descriptions are provided in Table 5, and the dataset is hosted on HuggingFace.The authors direct readers to supplementary code for how the dataset is used in evaluation.Source: §B.4 · p. 17, Table 5 · p. 17

C. Why timeit?

  • The benchmark uses Python’s timeit because repository maintainers use it in performance reports and asv uses it for timing.The Python API’s repeat() provides runtime distributions for mean and standard-deviation analysis; tasks with high measurement variance are filtered.Source: C. Why timeit? · pp. 17–18

D. Real-World Workload-Specific Optimization

  • Experts optimize for workloads that matter, sometimes accepting regressions on other inputs; SWE-FFICIENCY’s single-workload tasks reflect that practice.Examples include a NumPy change targeting small arrays despite added overhead on large arrays, and a pandas tradeoff favoring long DataFrames.Source: D. Real-World Workload-Specific… · p. 18

E. Additional Details on Data Collection Procedure

The collection pipeline filters performance-related repository changes, selects tests intersecting expert edits, manually annotates workloads, and validates correctness and speedups. This yields tasks with reproducible expert baselines and relevant correctness checks.

E. Additional Details on Data Collection Procedure

  • The pipeline scrapes 96,457 pull requests from nine open-source Python repositories, then selects performance-regression candidates using performance keywords, excluding test-modifying PRs and retaining only edits that meaningfully change the AST.Correctness tests and before/after performance workloads retain instances with statistically significant improvement; this yields 498 tasks across nine repositories.Source: E. Additional Details on Data… · pp. 18–19

E.1. Repo selection and instance scraping

  • The nine selected repositories are widely used Python projects in data science, machine learning, and high-performance computing, including NumPy and SciPy.The authors relax SWE-bench’s requirement for a linked issue because performance PRs may be opportunistic.Source: §E.1 · p. 19

E.2. Performance regression attribute filtering

  • Performance filtering excludes test-changing PRs, checks performance-related metadata, and requires meaningful AST edits; sampled SWE-bench candidates instead indicated new behavior.The authors report that all 50 manually reviewed candidates from the keyword filter had issue statements or hints indicating new behavior.Source: §E.2 · p. 20

E.3. Identifying covering correctness tests

  • Coverage and symbol expansion identify unit tests intersecting the expert diff, retaining only tasks with at least one guarding test.Across 498 tasks, the median is 7,723 covering tests per instance; over 90% have more than 100, and approximately 74% have more than 1,000.Source: §E.3 · p. 21

E.4. Annotating performance workloads

  • Workloads are manually reconstructed from PR descriptions, diffs, discussions, and linked issues because performance scripts are difficult to extract consistently.The authors report that PR code blocks vary in format, timing mechanism, and execution method; they leave LM-based extraction to future work.Source: §E.4 · p. 21

E.5. Execution-based filtering

  • Execution filtering retains workloads that run on both code versions and show statistically significant speedups, while correctness tests are run ten times to screen for flakiness.The speedup filter requires the pre-edit minus post-edit mean runtime to exceed twice the post-edit standard deviation.Source: §E.5 · p. 23

F. Techniques for Improving Performance Reproducibility

The benchmark reduces runtime-measurement variability by separating setup from timed execution and isolating container CPU and memory resources. Its reproducibility design combines prebuilt images with CPU affinity and per-container memory and NUMA controls.

F. Techniques for Improving Performance Reproducibility

  • The reproducibility strategy combines prebuilt containers and CPU pinning to reduce incidental variability in runtime and throughput measurements.Prebuilding avoids setup work on the critical path; CPU pinning separates container execution from Docker management.Source: F. Techniques for Improving… · p. 24

F.1. Prebuilding instance docker images

  • Prebuilt Docker images keep dependency installation and environment setup out of timed evaluations, reducing run-to-run variability and contention among concurrent workers.Timed CPU resources are reserved for the containerized program and harness.Source: §F.1 · p. 24

F.2. Pinning Containers and Docker Daemon to CPU Cores

  • CPU affinity assigns containers disjoint logical-core groups and reserves separate physical CPUs for Docker background services.Containers use four vCPUs and 16GB of memory, with memory bound to the NUMA node corresponding to their assigned cores.Source: §F.2 · p. 24

G. Agent Harness Prompts and Details

The appendix describes the prompts and harness settings used to evaluate agents optimizing repository code against supplied workloads. Agents must preserve functional correctness, improve workload performance through repository edits, and verify their changes.

G. Agent Harness Prompts and Details

  • The appendix provides prompt and harness details for generating the evaluation results, with further information in the code artifact.The prompt asks agents to optimize a specified workload in a repository using file utilities and bash execution in a containerized environment.Source: G. Agent Harness Prompts and Details · p. 24, §G.1 · p. 24

G.1. Code Optimization Task Prompts

  • Agents receive a repository and workload, and must improve the workload without changing its implementation, editing repository source instead.They are instructed to preserve functional equivalence and check relevant repository tests, while not needing to address tests that already fail.Source: §G.1 · p. 25
  • Agents must ensure the original workload runs successfully and improves, rebuild after non-Python edits, and use the preinstalled dependencies.The prompt cautions that all tests may take a long time, so agents should identify tests relevant to their changes.Source: §G.1 · p. 25
  • The prompt directs agents to explore the repository, reproduce the workload, inspect metrics, and iteratively edit and rerun repository code.Agents are asked to verify relevant test statuses and reconsider approaches when a change does not improve performance.Source: §G.1 · p. 25

H.1. Benchmark Performance Full Results

  • Table 6 defines speedup ratio as pass@1, normalizing each model's speedup by the gold patch's speedup and aggregating across tasks with a harmonic mean.Human-expert speedup ratio is 1.0×; the evaluation uses a three-hour wall-clock limit and 100 interaction turns for both harnesses.Source: Table 6 · p. 26, §H.1 · p. 26

G.2. Details on Language Model Sampling Parameters

  • SWE-agent runs have a $1 spending cap, 100-turn and time limits, while OpenHands runs enforce action-count and time limits.Both harnesses use recommended greedy sampling; GPT-5 MINI uses temperature 1 and other models temperature 0, as specified in the appendix.Source: §G.2 · p. 26

H.2. How does LM Performance Scale with Dataset Difficulty?

As tasks become more difficult, model speedup ratios decrease. The profiling appendix defines attribution and coverage metrics to examine how model edits compare with expert optimizations and where improvements occur in the workload call graph.

H.2. How does LM Performance Scale with Dataset Difficulty?

  • As tasks are included across higher difficulty levels—measured by workload duration, gold-patch size, or gold-patch speedup—aggregate model speedup ratio decreases.Figure 16 plots the trends across those three difficulty measures.Source: §H.2 · p. 26, Figure 16 · p. 28

I. Profiling-Based Attribution and Coverage Metrics

  • Function-level profiles record self and cumulative time, call counts, and caller–callee relationships before and after edits; metrics are computed only for correctness-passing instances.This profiling supports attribution of model and expert performance changes to functions.Source: §I · p. 28

I.1. Data and Notation

  • The notation defines per-function and workload-level improvements from pre- and post-edit profiles, and reports whole-trace speedups normalized by pre-edit workload time.Functions are identified by source file, line number, and name.Source: §I.1 · p. 28

I.2. Selecting Core Function-Level Improvements Without Double Counting Naively summing ∆cum

  • To avoid double-counting caller and callee gains, selection uses a deepest-first, non-overlapping set of improved functions; if none are selected, it includes the best candidate.Candidates are restricted to functions reachable from the workload and in files edited by the actor; a per-instance threshold filters small gains.Source: §I.2 · p. 29
  • The selection threshold is the larger of an absolute time floor and a fraction of the expert's end-to-end improvement, with the fraction defaulting to 0.02.Candidates must also exceed the threshold in cumulative improvement.Source: §I.2 · p. 29

I.3. Expert-Relative Coverage (ERC) and Loss Decomposition

  • Expert-relative coverage measures whether model edits reach the expert's optimized files and functions; missed expert mass is split into wrong-file and within-file localization losses.ERCfile captures file coverage, while ERCfunc captures function coverage within those files.Source: §I.3 · pp. 29–30

I.4. Edited-File Overlap

  • Edited-file overlap is measured with Jaccard similarity; it is omitted when neither actor meets the selection threshold and the union is empty.Source: §I.4 · p. 30

I.5. Depth of Optimization from the Workload

  • Depth statistics describe where selected improvements lie in the workload call graph and how much selected mass is reachable from the workload.Selection uses cumulative improvement to capture inclusive gains, while depth weights use total improvement to avoid double-counting; the call graph is taken from the pre-edit trace, and statistics use correctness-passing instances.Source: §I.5 · p. 30

I.6. Full Speedup Attribution Results

This section presents full speedup attribution results alongside a transformed, memory-isolated pandas workload and its timing harness. It compares LM-generated edits with expert changes and describes controls against timing-aware and cross-run cache-based reward hacking.

I.6. Full Speedup Attribution Results

  • WrongFileLoss is roughly 37%−45% across models (mean ≈41.5%), InFileLoss roughly 27%−33% (mean ≈29.7%), and their sum implies wrong-function choices about 69%−73% (mean ≈71.2%).These metrics cover LM patches that pass correctness and speed up the pre-edit runtime, but need not beat the expert; ERCfunc is ≈0.26−0.31. Experts typically optimize at call-stack depth 4.5−4.9, versus 3.8−4.2 for LLMs.Source: Table 9 · p. 30, §I.6 · p. 30, Table 10 · p. 31

J. Comparison of LM Generated Edits Versus Experts

  • The appendix supplies raw, commented diffs for the expert-versus-LM edit comparisons, alongside additional comparison material.The main text omits comments and surrounding lines to focus on changed lines.Source: J. Comparison of LM Generated Edits… · p. 31

K. Synthetically Generating Performance Workloads

  • The workload-generation pilot used GEMINI 2.5 FLASH with the gold diff and pre-edit source context, asking for self-contained, representative Python workloads with repeated timing.The instructions aimed to match human annotators’ workload instructions so that the comparison isolates workload-design quality rather than interface differences.Source: §2 · pp. 31–32, K. Synthetically Generating… · p. 31

2. The concatenated pre-edit files for those paths

  • In 76% of cases, manual annotations show a larger performance delta on the expert patch than LLM-generated workloads, and 47% of workloads show no significant delta.The comparison holds the gold patch fixed and evaluates both workloads on the same base and patched code states.Source: §2 · p. 33
  • The pilot is a lower bound: the model had medium capability and lacked PR descriptions and discussion threads available to annotators, so the authors expect the gap may narrow with stronger models and matched context.The authors retain manual annotation as the most reliable current choice and identify LM-assisted workload generation with PR context and a verification harness as future work.Source: §2 · p. 33

L.1. Preventing Stack-Frame–Based Reward Hacking

  • An AST-based checker flags newly added stack-introspection logic that can detect timed execution, while permitting input-attribute-based tuning and passing expert patches without false positives.The checker is diff-relative and resolves aliases and dynamic imports; the authors report that it rejects previously identified stack-frame exploits and does not flag gold edits.Source: §L.1 · pp. 33–34
  • Fork-per-run workload execution prevents module-level caches and mutated globals from persisting between repetitions, helping ensure measured speedups reflect algorithmic improvements rather than residual state.The transformation preserves workload logic, result slicing, and summary behavior while running each repetition in a fresh child process.Source: §L.2 · pp. 34–35
  • The pandas workload is timed in forked child processes, with 1,000 repeats of five workload executions and mean and standard deviation reported.The workload uses a 100,000-element nullable Int32 array with randomly placed missing values.Source: §L.2 · pp. 36–37

M. Full Example Diffs of LM Generated Edits Versus Experts

  • Full raw diffs with inline comments for Figures 8 and 9 appear as Figures 17 and 18, alongside an additional diff for 19 comparison.The main text removes comments and surrounding lines to focus on changes between expert and LM-generated diffs.Source: M. Full Example Diffs of LM Generated… · p. 37

N. Speedup Ratio Stability Analysis and Error Bars

The section tests whether measurement variability affects Speedup Ratio (SR) assessments. Across three independent runs, aggregate SR was stable, with final-score variance below 0.5%.

N. Speedup Ratio Stability Analysis and Error Bars

Loading 2511.06090v3…