Source-linked AI summary
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Zhi Chen, Zhensu Sun, Yuling Shi, David Lo, Lingxiao Jiang
TL;DR
Repository-level performance-optimization benchmarks are increasingly used to assess coding agents, but their scores may reflect unstable runtime signals, scoring rules, and task coverage. We audit three benchmarks through cross-machine reference-patch replays, scoring-rule comparisons, and task-level analysis. The results show limited cross-machine validity, rank changes under alternative scoring, and broad public coverage of replay-valid tasks.
Problem
Benchmark scores combine runtime measurements, scoring rules, and task coverage, leaving the reliability of their evidence about coding-agent progress open to audit.
Method
The study replays 740 official reference patches across four machines, audits scoring rules using public outputs, and evaluates coverage across ten submissions per task.
Results
Across replay-valid GSO and SWE-fficiency tasks, at least one public submission matched or beat the reference patch on 384/450 tasks, while 449/450 beat the base program.
Takeaways & Limitations
Interpreting benchmark progress requires separating stable performance signals, scoring-rule effects, and task-level gaps from aggregate leaderboard rankings.
Takeaways & Limitations
The findings cover three benchmarks and specific public leaderboard snapshots, and the ten-submission analysis is a task-coverage proxy rather than a newly engineered multi-agent workflow.
Abstract
from arXiv · showhide
Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.
I. INTRODUCTION
Repository-level performance benchmarks evaluate executable optimizations, but their measurements and aggregate scores can be unstable or difficult to interpret. This audit replays reference patches, compares scoring rules, and examines task-level coverage to identify where leaderboard evidence is reliable.
- Motivation: Performance benchmarks compare base, reference, and submitted patches using noisy runtime measurements rather than fixed pass/fail outcomes.Runtime varies with scheduling, cache state, memory contention, and machine microarchitecture.
- Reference-patch validity: Only 39/102 GSO, 11/140 SWE-Perf, and 411/498 SWE-fficiency tasks remained valid in every cross-machine replay.SWE-Perf was especially fragile because many reference patches produced close-to-zero runtime changes.
- Leaderboard scoring: Official rankings disagreed on 9 of 28 pairwise comparisons among eight submissions shared by GSO and SWE-fficiency.SWE-fficiency’s worst ten tasks carried 58.5–82.8% of score weight, showing how scoring rules can reshape rankings.
- Task-level coverage: Across 450 replay-valid GSO and SWE-fficiency tasks, at least one public submission matched or beat the reference patch on 384 tasks and beat the base program on 449.The remaining gap was mainly reaching reference-patch speed rather than producing any correct runtime improvement.
- Implications: Leaderboard scores alone are insufficient evidence of agent capability without examining reference-signal stability, scoring-rule sensitivity, and unsolved-task contributions.The audit proposes complementing aggregate rankings with these task-level diagnostics.
- Reference-patch validity: 740 official reference patches were replayed across four Google Cloud machine types and three rounds using each benchmark’s validity rules.The audit covers GSO, SWE-Perf, and SWE-fficiency.
II. STUDY DESIGN
The study selects three recent repository-level performance-optimization benchmarks and frames three design questions about machine robustness, scoring sensitivity, and task-level solvability. It focuses on executable, reviewed benchmark artifacts intended to evaluate agents’ performance-engineering ability.
- Benchmark Selection: The study includes GSO, SWE-Perf, and SWE-fficiency because they evaluate repository-level runtime optimization with executable workloads or tests.Selection excludes efficiency benchmarks that are function-level, code-only, unpublished, or leaderboard-only.
- Research Questions: RQ1 asks whether official reference patches remain valid when replayed across different machine configurations.The question tests the machine-agnostic assumption that reference patches and workloads preserve their performance signal.
- Research Questions: RQ2 asks whether the eight submissions shared by GSO and SWE-fficiency receive similar rankings under the two benchmarks’ scoring rules.The analysis keeps task outputs fixed while changing the aggregation rule.
- Research Questions: RQ3 examines task-level outputs because different public submissions may solve different tasks, making a single leaderboard entry an incomplete boundary of task solvability.This motivates inspecting coverage across submissions rather than relying only on aggregate ranks.
III. RQ1: CROSS-MACHINE REFERENCE-PATCH VALIDITY
The audit replays official reference patches across four Google Cloud machine profiles and twelve machine-round combinations, preserving each benchmark’s workloads and validity rules. Most tasks remain executable, but portability failures arise from dependencies, external resources, and CPU-specific instructions.
- Experiment design: 740 official reference patches are replayed across four Google Cloud machine profiles and twelve machine-round combinations.The replay keeps the cloud provider and resource configuration fixed while varying processor vendors and generations.
- Validity checks: The replay tests both whether reference patches beat the base program in every machine-round and whether they satisfy each benchmark’s original construction rule.The original-rule check is stricter than the faster-than-base check.
- Replay evaluability: 4/102 GSO, 2/140 SWE-Perf, and 0/498 SWE-fficiency tasks are not fully evaluable across all replay machines.Complete-by-machine counts require all three rounds on each of the four replay machines.
- Replay evaluability: GSO replay failures include functional-equivalence, external-data, image-decoding, and x86-64 portability failures, while SWE-Perf failures involve missing dependencies.SWE-fficiency has no replay-evaluability failure under the replay setup.
C. RQ1.2 Reference Validity Under Replay
Cross-machine replay shows that runnable reference patches do not necessarily preserve benchmark-valid performance signals. SWE-Perf is especially fragile because its reference patches usually produce runtime changes too close to zero for small timing shifts to preserve statistical support.
- Reference validity: 91/102 GSO, 48/140 SWE-Perf, and 470/498 SWE-fficiency reference patches run faster than the base code across machines.Applying each benchmark’s original validity rule leaves only 39, 11, and 411 valid tasks, respectively.
- Measurement: Negative runtime changes indicate faster reference patches, and for GSO and SWE-fficiency speedup s is converted as 1/s−1.A 1.20× speedup corresponds to a −16.7% runtime change.
- Signal size: -0.03% is the median SWE-Perf runtime change, making small timing shifts sufficient to cross zero or lose statistical support.101/138 SWE-Perf tasks have median changes within five percentage points of zero.
E. RQ1 Summary
The audit separates replay-validity failures from the way benchmark outputs become leaderboard scores. It also frames task-level public-submission coverage as distinct from whether any submission reaches reference-patch speed.
- RQ1 summary: Most reference patches are runnable, but runnable does not imply benchmark-valid under cross-machine replay.Reapplying the original construction rules leaves 39 GSO, 11 SWE-Perf, and 411 SWE-fficiency tasks valid.
- Audit scope: The audit distinguishes what submitted patches achieve on individual tasks from how benchmark scoring rules weight those outcomes into leaderboard scores.The scoring analysis uses GSO and SWE-fficiency because they released public ranking data; SWE-Perf did not provide comparable public agent-output data.
- Scoring rules: A reference-level success is a correct submitted patch whose speedup matches or exceeds the official reference patch.GSO counts such successes with a binary gate: below-reference patches receive no partial credit.
- Scoring rules: SWE-fficiency scores each task by SpeedUp Ratio relative to the reference patch and aggregates task values with a floored harmonic mean.Values below 1 indicate slower submissions; values above 1 indicate submissions that beat the reference patch.
- Scoring rules: SWE-fficiency’s penalty can make a floor-level task outweigh a full reference-matching denominator.At the official floor f = 0.001, one task adds 999 denominator units.
B. RQ2.1 Cross-Benchmark Metric Sensitivity
The eight shared submissions receive different rankings across GSO and SWE-fficiency, and rescoring shows that aggregation rules explain part of the disagreement.
- Official comparison: 9 of 28 pairwise submission orders disagree between the official GSO and SWE-fficiency leaderboards.The comparison covers eight public submissions shared by both benchmarks.
- Rescoring diagnostics: 0.452 to 0.762: rescoring SWE-fficiency outputs with a GSO-style reference-level gate raises Spearman rank correlation with GSO.Discordant pairs decrease from 9 to 6 under this rescoring.
- Rescoring diagnostics: 0.238: applying SWE-fficiency’s harmonic scoring to available GSO outputs produces weak agreement with the official SWE-fficiency ranking.This direction flips 11/28 head-to-head comparisons.
- Interpretation: Scoring rules affect submission ordering, while task sets and per-task outputs also contribute to the final rankings.The rescoring analyses are treated as diagnostics rather than replacement rankings.
C. RQ2.2 Low-Speedup Tail Dominance
SWE-fficiency’s harmonic scoring gives disproportionate influence to the lowest-speedup tasks, making aggregate scores sensitive to a small tail of failures.
- Mechanism: Very low-SR tasks have greater leverage because each enters the harmonic-mean denominator as 1/max(SR, 0.001).The analysis measures leverage by sorting tasks by denominator share within each submission.
- Tail weights: 58.5–82.8%: the worst ten tasks carry this share of the official score denominator.Worst-1 tasks carry 6.3–33.6%, while worst-5 tasks carry 31.4–73.1%.
- Interpretation: When ten tasks carry more than half the denominator, rank differences may reflect a few near-floor failures rather than broad benchmark performance.The authors identify this as an interpretability issue, not an argument to ignore severe slowdowns.
D. RQ2.3 Exploring Reasonable Penalty Bounds
A bounded-penalty diagnostic tests how limiting the influence of one bad task changes SWE-fficiency rankings without claiming that its chosen cap is optimal.
- Interpretation: The leaderboard order depends on the penalty budget assigned to one bad task.The diagnostic keeps harmonic aggregation while limiting the largest below-reference penalty.
- Results: 6/8 submission ranks move and 8/28 pairwise orders flip under the bounded-penalty diagnostic.The comparison is against the official SWE-fficiency ranking.
- Results: The cap moves Claude Opus 4.6 and GPT-5.2 above Claude Opus 4.5, while GPT-5 drops.The resulting ranks become closer to median SR and above-reference task counts.
- Interpretation: The authors do not claim that f = 0.5 is the correct constant; the diagnostic instead exposes sensitivity to penalty design.Changing the single-task penalty cap reshuffles the leaderboard.
E. RQ2 Summary
Leaderboard rankings combine task-level submission outcomes with benchmark scoring choices, while multi-submission coverage reveals that many replay-valid tasks already have strong public solutions. The remaining tasks are concentrated at the stricter reference-level performance gate rather than basic correctness or base-program speedup.
- Cross-benchmark rankings: 9 of 28 pairwise orders disagree between the official GSO and SWE-fficiency rankings for eight shared submissions.Rescoring the same task outputs changes ranks, showing that both task outcomes and scoring design shape the final order.
- Task coverage: 449/450 tasks have a passing public patch that beats the unmodified base program, and all 450 have at least one passing public patch.Only 66 tasks remain below the reference-level gate.
- Remaining gaps: The remaining performance gaps usually persist after correctness and base-program speedup are achieved.The best public patch often passes tests and improves runtime but still trails the reference patch.
- Task coverage: 384/450 replay-valid tasks have at least one of 10 public submissions matching or beating the reference patch.The analysis considers 39 GSO and 411 SWE-fficiency tasks; SWE-Perf is excluded because it has only 11 replay-valid tasks and no comparable released public solution data.
B. RQ3.2 Are the Remaining Tasks Truly Unsolved?
The remaining tasks usually have correct, runtime-improving public patches that nevertheless trail the reference patch. Strategy mismatch is associated with a modest median gap but does not explain most remaining shortfalls.
- Remaining-task performance: 27 SWE-fficiency tasks reach 90–100% of reference speedup, but 8 reach only 10–50% and one falls below 10%.The tail contains substantial gaps even though most remaining tasks are near the reference.
- Remaining-task performance: 65/66 best public patches beat the base program, while their median speedup reaches 85.3% for GSO and 87.9% for SWE-fficiency relative to the reference.Every task has a correct public patch; only one passes tests without beating the base program.
- Strategy alignment: 32/66 best public patches match the reference’s high-level category, while 34/66 use a different category or show no visible production optimization.Algorithm references are matched most often at 21/31, whereas memory-heavy references are approached differently in 9/12 cases.
- Strategy alignment: Same-category patches reach 89.8% of reference speedup versus 81.1% for different-category patches, but the groups overlap substantially.The remaining multipliers are 1.12× versus 1.23×; 16/32 same-category cases remain below 90%, while 11/32 different-category cases reach 90–100%.
- Strategy alignment: 32 same-category patches remain slower than the reference, while 11 different-category patches reach 90–100% of reference speedup.Thus, optimization strategy mismatch is not the main reason tasks remain below reference.
D. RQ3 Summary
Across replay-valid GSO and SWE-fficiency tasks, public submissions already cover most tasks, and the remaining cases generally involve closing a speed gap rather than achieving correctness. The discussion therefore recommends interpreting leaderboard and multi-agent results at the task level and expanding benchmarks toward fuller performance-engineering workflows.
- RQ3 Summary: 384/450 tasks have a public submission matching or beating the reference patch, while all 450 have a passing patch and 449 beat the base program.The remaining 66 tasks are therefore usually not basic correctness failures.
- Implications for benchmark users: Leaderboard ranks should not be read as direct labels of agent capability because cross-machine instability and aggregation rules affect benchmark signals.Reports should distinguish verified from unstable tasks, show per-task score weight, and compare use-case-appropriate aggregation rules.
- Implications for agent builders: For multi-agent evaluation, the narrow remaining room means measured gains may depend heavily on a small set of tasks.These benchmarks remain useful for studying the final step from faster-than-base to reference-level speed, but are less suitable as sole evidence of general performance-engineering ability.
- Implications for benchmark designers: Future benchmarks should add hotspot localization, ambiguous targets, partially hidden validation workloads, and resource metrics beyond a single runtime number.The proposed workflow includes reasoning from profiles or traces and evaluating CPU time, latency, allocation behavior, memory footprint, and regression risk.
VII. THREATS TO VALIDITY
The audit uses a conservative replay-validity rule and covers only three recent benchmarks and specific public snapshots. Its conclusion is that benchmark interpretation should account for unstable signals, task coverage, and the limits of patching pre-specified workloads.
- Internal validity: A task is replay-valid only when its reference patch satisfies the original construction rule in all 12 machine-round replays.This conservative rule may undercount usable tasks but protects later analyses from unstable reference signals.
- External validity: The findings cover three recent repository-level benchmarks and specific public leaderboard snapshots, not all coding-agent or performance-engineering settings.The 10-submission analysis is a task-coverage proxy rather than a newly engineered multi-agent workflow or claim about one configuration.
- Data construction validity: SWE-Perf is excluded from metric and RQ3 task analyses because comparable public agent-output artifacts were unavailable.The study relies on each benchmark’s released tasks, reference patches, public submissions, and scoring records.
- Scope: The audit does not introduce another benchmark or new agent leaderboard; it evaluates replay reliability, scoring effects, and task-level gaps in existing artifacts.This positions the work as an audit of current benchmark evidence rather than a replacement benchmark.
- Conclusion: Future benchmarks should test reasoning from performance symptoms or profiles, target selection, and validation of runtime alongside memory, latency, and other resource costs.The proposed direction moves beyond patching a pre-specified entry point under a fixed workload.