PaperlayerView exact revision on arXiv

Source-linked AI summary

HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills

Yakup Koray Budanaz, Lorenzo Paleari, Marcin Copik, Alexandru Calotoiu, Timo Schneider, Benjamin Chen Ming Choong, Shriram Chandran, Lars Hulsbergen, Levi Widmer, Torsten Hoefler

arXiv:2610.20673v1cs.SE

TL;DR

Existing coding-agent benchmarks cover limited hardware, languages, or application domains, and measurements can be vulnerable to unreliable timing and evaluation loopholes. HPCAgent-Bench evaluates 689 kernels with controlled setup changes, hidden-input judging, and explicit token-cost metrics. Across broad settings, agents beat automatic parallelizers by up to 4.2×, while most non-device-specific answers remain correct on another machine.

  • Problem

    Existing benchmarks target one device, restricted languages or models, or a single domain, while reported speedups can be unreliable and evaluations vulnerable to exploits.

    Source: §I · p. 1

  • Method

    HPCAgent-Bench evaluates 689 kernels across three tracks, using controlled one-component interventions, hidden fuzzed-input judging, and metrics for intervention efficacy and weighted token cost.

    Source: §I · pp. 1–2

  • Results

    Open-weight models beat automatic parallelizers by up to 4.2×, and 93–99% of answers without device-specific code remain correct on a second machine.

    Source: C. Portability: optimizations across… · p. 6, §VI · p. 8

  • Takeaways & Limitations

    The benchmark supports comparisons of agent setups by changing one component at a time and reporting speedup alongside token cost.

    Source: §I · p. 1

Abstract

from arXiv · show

Coding agents are increasingly used to optimize code, but existing benchmarks fix one accelerator, programming model or harness, so they cannot isolate what each component contributes or show whether added skills and tools repay their cost. We present HPCAgent-Bench with 689 kernels in three tracks: loop-level optimization challenges, kernels extracted from HPC codes, and machine-learning models. The framework specifies each kernel once in NumPy, derives C and Fortran references, accepts submissions through a common C ABI or Python interface, and runs them on CPUs or GPUs, on one or multiple nodes. An external judge checks correctness on hidden fuzzed inputs and times each submission, closing paths agents use to exploit the testing system. We introduce two metrics for reporting results: \textit{intervention efficacy} measures an intervention, such as adding a tool, through paired runs on speedup, success rate and cost, and \textit{weighted token cost} reports input, cached-input and output tokens under explicit weights. Agents reach up to $4.2\times$ speedup over automatic parallelizers, and their optimizations transfer to another machine: $93$--$99\%$ of answers without device-specific code remain correct, and per-kernel speedups keep a rank correlation of $r_s\ge0.84$. Across 40 matched setups, interventions significantly change speedup only twice but token cost by up to $3.0\times$. Because token weighting decides whether a saving is significant, we state the weights and release every token count as a standard for cost reporting. We provide the first comprehensive, broad and hardened benchmark for the sound evaluation of agentic optimization, with transparent cost metrics.

I. INTRODUCTION

Existing agent benchmarks cover limited hardware, programming models, or domains, while performance measurements can be unreliable or vulnerable to evaluation exploits. HPCAgent-Bench combines a broad kernel corpus with controlled setup changes, hardened grading, and explicit cost metrics.

I. INTRODUCTION

  • Existing benchmarks are narrow in hardware, languages, or domains, and reported speedups can be unreliable because of retiming failures, cross-machine losses, and evaluation exploits.None spans low-level and parallelism challenges, production HPC code, and ML models.Source: §I · p. 1
  • HPCAgent-Bench compares setups by changing exactly one component and uses hidden fuzzed inputs and an external judge to check correctness and time submissions.A speedup counts only when statistically significant; fixed container images and isolated execution further constrain evaluation.Source: §I · p. 1
  • Its 689 kernels span loop-level reasoning, scientific-computing kernels from production codes and suites, and machine-learning operators and models.The corpus combines HPC proxy applications, loop challenges, and ML models under one ABI for CPU and GPU execution.Source: §I · pp. 1–2
  • Intervention efficacy attributes changes in success, speedup, and token cost to one intervention, while weighted token cost reports input, cached-input, and output tokens.The stated metrics are intended to make cost conventions explicit.Source: §I · p. 1

II. DESIGN

  • A kernel’s NumPy specification generates C and Fortran references, bindings, and prompts; each episode accepts one artifact through a shared C-ABI symbol, source file, or Python binding.Run configuration fixes kernel selection, language, submission mode, precision, array residency, and optimizer; residency sets the timing boundary.Source: §II · p. 2, A. What a run declares · p. 2

A. What a run declares

  • Scaling experiments support one or multiple nodes: weak scaling grows declared sizes with processing elements, while strong scaling keeps total work constant.The evaluation covers three open-weight models, two devices, and a second machine.Source: A. What a run declares · p. 2, §I · p. 2

B. Inputs, secrecy, and verification

The framework uses secret-seeded fuzzed inputs to grade correctness and separates candidate scoring from final submission. Its corpus draws on loop challenges, production scientific-computing codes, HPC suites, and machine-learning models.

B. Inputs, secrecy, and verification

  • Input draws vary declared sizes and cycle array contents through uniform, normal, and lognormal distributions, subject to each kernel’s constraints.Kernels with restricted value domains can declare a domain or custom initializer.Source: B. Inputs, secrecy, and verification · p. 2
  • Separate secret seeds generate score-tool and final-submission inputs, preventing agents from optimizing specifically for inputs they have already observed.The score and submission routes use independent fuzzed draws.Source: B. Inputs, secrecy, and verification · p. 2
  • Correctness is all-or-nothing: integer and boolean outputs must match exactly, while floating-point outputs use precision-specific absolute and relative tolerances.The absolute tolerance accounts for reordered sums and accumulation precision.Source: B. Inputs, secrecy, and verification · pp. 2–3

C. Corpus sources

  • Scientific-computing kernels come from production codes and HPC proxy applications, while the corpus also includes modernized loop challenges and machine-learning models.Scientific-computing kernels are organized by the thirteen Berkeley dwarfs.Source: C. Corpus sources · p. 3

III. SCORING

The scoring protocol measures statistically supported speedups, parallel scaling, intervention effects, and token costs. It aggregates results while accounting for unsolved tasks and the cost of failed runs.

Speedup score and aggregation

  • A task’s speedup score is the geometric mean of per-input speedups, with an unsolved task receiving no score.Speedups are credited only when a one-sided Mann–Whitney U test finds a significant difference; confirmed slowdowns score below 1, otherwise the input scores 1.Source: §III · p. 3
  • Benchmark runs report success rate and the geometric mean speedup over solved tasks, and compare runtimes with a rank test.Token and agent-time limits are per episode; episodes stopped by either limit are rerun once with 2–4× the limit.Source: §III · p. 3, Table II · p. 4

Scaling score

  • Scaling efficiency compares an answer’s runtime on P processing elements with the baseline’s one-element runtime, adjusted for problem growth.Efficiency of 1 matches ideal scaling of the baseline; above 1 means the answer beats the baseline per processing element.Source: §III · p. 3

Intervention efficacy score

  • Intervention efficacy compares paired runs with and without an intervention using success, speedup on jointly solved kernels, and token-cost ratios.Speedup uses only kernels solved by both runs, while cost includes every kernel because failed tasks still consume tokens.Source: §III · p. 3

Token cost

  • Weighted token cost combines new input, cached input, and output tokens, with billed weights of 1, 0.1, and 1, respectively.The billed weighting is the default; alternative weightings count cached input at zero or at the full input rate.Source: §III · p. 4

IV. EVALUATION

The evaluation tests whether HPCAgent-Bench is valid, portable, controlled, general, and transparent. It describes the experimental setup and the safeguards used to make reported speedups trustworthy.

IV. EVALUATION

  • The evaluation asks whether speedups are trustworthy, transfer across machines, respond to controlled setup changes, extend to accelerators and distributed computing, and can be traced to code.Setups fix the model, language, device, skills, and harness; costs use billed weights and results use 95% log-t intervals with Benjamini–Hochberg correction.Source: §IV · p. 4, A. Experimental setup · p. 4

A. Experimental setup

  • In the matching kernel setup, speedups span 0.51–100×; at temperature 0, the spread narrows to 1.8–11× because only two runs use threads, but remains.Figure 2 compares 20 independent runs per model across three kernels at temperatures 0 and 1, showing speedup distributions, billed token cost per run, and tasks completed.Source: A. Experimental setup · p. 4, Figure 2 · p. 4

B. Validity: speedups you can trust

  • Secret, changing inputs, sandbox checks, plausibility bounds, and manual review harden grading against caching, timing, and other cheating attempts.With fixed inputs, one TSVC s311 answer scored 5309× by caching; on new draws it scored 20×.Source: B. Validity: speedups you can trust · p. 5
  • The runtime-distance recurrence has K independent chains: K = 0 needs no ordering, K = 1 needs a scan, and large K exposes multiple scan chains.Kimi-K2.7’s tiled parallel scan reaches 34.6×; a Qwen3.8-27B attempt to infer hidden inputs through timing scored about 1× in the final grade.Source: B. Validity: speedups you can trust · p. 5, Figure 4 · p. 5

C. Portability: optimizations across machines

The paper tests whether optimized answers transfer to a second machine, then changes one setup component at a time to measure its effects on performance, success, and cost.

D. Control: one intervention at a time

C. Portability: optimizations across machines

  • On GH200, 93–99% of non-device-specific answers remain correct, per-kernel speedups retain rs ≥0.84, and C gains 1.6× over Numba.The best MI300A answer is also best on GH200 for 42% of kernel-language pairs and reaches at least 90% of GH200’s best speedup for 82%.Source: C. Portability: optimizations across… · p. 6, Table IV · p. 6

D. Control: one intervention at a time

  • The Language skill packet rarely changes speedup; the performance toolkit changes neither speedup nor cost on 35 scientific-computing kernels.The exception is OSS-120B in C, with ρS = 0.90; GPU skill-packet setups also raise billed cost for some models.Source: D. Control: one intervention at a time · p. 6
  • CPF halves Qwen3.8-27B’s billed tokens at unchanged speedup and raises OSS-120B’s speed; Terse changes neither significantly but solves 2–5 fewer kernels per setup.Kimi-K2.7 shows no change with CPF.Source: D. Control: one intervention at a time · p. 6
  • Repository framing changes no measured outcome; harnesses also leave speedup unchanged, though OpenHands and mini-SWE-agent reduce billed tokens in specific setups.AutoKernel raises OSS-120B’s cost 1.3× while its speedup stays within noise.Source: D. Control: one intervention at a time · pp. 6–7
  • Token weighting changes the measured cost effect up to threefold: OpenHands saves Qwen3.8-27B ρC = 1.10 by effective cost but 3.2× by total cost.The difference reflects whether repeated cached context is charged.Source: Figure 7 · p. 7, D. Control: one intervention at a time · p. 7

E. Generality: accelerators and distributed computing

The benchmark evaluates agent code across languages, accelerators, and multi-node settings, and keeps submitted code and optimization histories inspectable. Results include gains over compiler comparators and distributed scaling on selected models.

E. Generality: accelerators and distributed computing

  • On kernels compiled by Pluto or PPCG, agents are 2.9–4.2× faster than Pluto and 1.8–3.4× faster than PPCG; HIP answers also outperform OpenMP target answers at similar solved counts.Pluto solves 32 of 40 kernels at 1.55× over Numba, while PPCG solves 16 at 9.2×.Source: E. Generality: accelerators and… · p. 7
  • On 16 GPUs across four nodes, Qwen3.8-27B reaches 10.0× single-GPU PyTorch and Kimi-K2.7 reaches 5.7×, above PyTorch distributed at 3.9×; OSS-120B stays below 1×.On the MoE router, Qwen3.8-27B’s fused one-pass implementation runs 121× faster than PyTorch on 16 GPUs.Source: E. Generality: accelerators and… · p. 7

F. Transparency: inspectable answers

  • The judge stores candidates and answers so readers can inspect the code behind each speedup; histories show agents refining answers through small edits across candidates.For ext_break_capture, a window-first search skips the first 40% of the array and is correct on every input produced by the supplied generator.Source: F. Transparency: inspectable answers · pp. 7–8
  • Seven of nine multicandidate episodes that end with window-first search begin with a plain parallel search and add the window in a later candidate.Final answers use window-first search in 7 of 12 Qwen3.8-27B setups, 5 of 9 Kimi-K2.7 setups, and none of 15 OSS-120B setups.Source: Figure 8 · p. 8, F. Transparency: inspectable answers · p. 8

V. RELATED WORK

Prior benchmarks tend to constrain task granularity, programming model, or hardware, while measurement and cost practices vary. The conclusion presents HPCAgent-Bench as a broad, controlled evaluation framework and reports the authors’ use of generative AI tools.

V. RELATED WORK

  • Related benchmarks cover GPU kernels, HPC codes, or repository-level performance tasks, but each fixes a programming model, task granularity, or hardware target.Of sixteen surveyed benchmarks, five publish versioned releases and eight a Hugging Face dataset.Source: §V · p. 8
  • Prior evaluations report unreliable speedups, loopholes, and exposed tests; cost reporting varies, so costs from different papers cannot be compared.A replay audit found reference patches retained their signal on only 39 of 102 GSO tasks and 11 of 140 SWE-Perf tasks.Source: §V · p. 8

VI. CONCLUSION

  • Across CPU loops, scientific codes, and distributed ML, open-weight models beat automatic parallelizers by up to 4.2×; among 40 intervention comparisons, only two significantly change speedup.Interventions change token cost by up to 3.0×, and the conclusion notes that repeated runs of one agent can span two orders of magnitude in speedup on one kernel.Source: §VI · p. 8

AI use statement

  • The authors used generative AI for implementation, related-work search and summaries, drafting, and editing, but not to design experiments or interpret results.They state that authors reviewed AI-assisted work and checked claims and numbers against sources and measurement databases.Source: AI use statement · pp. 8–9

APPENDIX

The appendix describes the benchmark corpus and experimental baselines, then specifies setups for loop-level, harness-intervention, and multi-GPU evaluations.

A. Platform and baselines

  • LLR uses Numba as its baseline, while SC uses the fastest among autoparallelized C, sequential C, and parallel Numba; ML uses single-GPU PyTorch.Pluto and PPCG are LLR comparators, not baselines.Source: A. Platform and baselines · p. 13
  • LLR-Focus40 evaluates GPU answers with device-resident arrays against CPU Numba, and its blind variant removes the score tool and allows one submission.The blind setup is paired with open-submission setups across C and Fortran for all three models, and HIP for Qwen3.8-27B and OSS-120B.Source: A. Platform and baselines · pp. 13–14

APPENDIX

  • 689 kernels span loop-level reasoning, scientific computing, and machine learning, with levels distinguishing primitives from fused, data-dependent, and application-scale work.The corpus includes 74 TSVC-2.5 extensions and 24 adversarial and canonicalization kernels beyond TSVC-2, plus additional ML kernels beyond KernelBench.Source: Table V · p. 14
  • Scientific-computing kernels are drawn from 64 applications and proxy applications, using hotspots while excluding MPI halo exchanges and collectives.The extracted kernels come from single-node compute hotspots.Source: APPENDIX · p. 13, Table VI · p. 14

A. Platform and baselines

  • Harness20 evaluates 20 C CPU kernels, with one episode per kernel, comparing Claude Code (control) against mini-SWE-agent, OpenHands, and AutoKernel on Qwen3.8-27B and OSS-120B.Kernels whose NumPy reference calls BLAS are excluded because speedups against their C baselines would measure library linking.Source: A. Platform and baselines · pp. 14–15
  • MLScale grades 19 bf16 models in single-submission mode under strong and weak scaling, while episodes have token and wall-time budgets.An episode that reaches either limit without an answer is rerun once with 2–4× the tokens and up to 4× the wall time.Source: A. Platform and baselines · p. 15

B. Second platform: GH200

The GH200 regrade tests portability across architectures, while the judging protocol uses secret inputs and timing checks to guard evaluation integrity.

B. Second platform: GH200

  • The GH200 regrade uses one module with 72 Grace cores and one H100, retimes the parallel Numba baseline there, and skips answers that fail stated architecture-portability checks.The checks cover x86-only constructs, AMD-only HIP features, and AMD-specific Triton options, cache hints, or runtime calls.Source: B. Second platform: GH200 · p. 15
  • On GH200, geometric-mean per-kernel speedups over Numba are 5.3, 7.5, and 4.9× for CPU; 38.7, 32.3, and 16.6× for OpenMP target offloading; and 54.5, 55.6, and 21.6× for HIP.The three values in each group correspond to Qwen3.8-27B, Kimi-K2.7, and OSS-120B, respectively; nonportable answers are skipped rather than graded wrong.Source: B. Second platform: GH200 · p. 15, Figure 9 · p. 16

C. Secrecy, timing and evasion

  • Secret score and submission seeds draw distinct tool and final-grade inputs, and an image check verifies neither seed is held in an agent image.Agents run unprivileged in private namespaces and see only their workspace, kernel, and read-only prompt and skill files.Source: C. Secrecy, timing and evasion · p. 16
  • Timed repetitions cycle through four seeded input draws in the same order for both sides, and the judge clock stops after device and OpenMP runtimes synchronize.The warmup uses the first draw, so a result cached from a prior call fails rechecking.Source: C. Secrecy, timing and evasion · p. 16
  • A one-sided Mann–Whitney test at α = 0.1 credits confirmed speedups and slowdowns, while unconfirmed inputs receive 1×; implausible timings can also be excluded.Inputs above the stated host or device speedup thresholds, below the declared-bytes/HBM timing bound, or triggering a device check are suspect.Source: C. Secrecy, timing and evasion · p. 16

D. Correctness

Correctness bands account for floating-point reassociation, and the scoring rules define paired comparisons, scaling measures, and token-count conventions.

D. Correctness

  • Floating-point outputs must meet elementwise absolute-plus-relative tolerance, with atol raised to a reassociation floor based on accumulation precision and reduction length.NaN and ±∞ positions must agree; configurations with εaccℓ ≥ rtol are ungradeable rather than incorrect.Source: D. Correctness · p. 18
  • Determinism requires two runs of one binary to differ by less than 30 times εℓ times the first run’s max-norm.The threshold is identified as LAPACK’s test-ratio threshold.Source: D. Correctness · p. 18

E. Scoring rules

  • Efficacy statistics use per-kernel log changes and paired comparisons; the reported interval assumes normal log-ratios across kernels, while runtimes use rank tests.No intervals are reported below six pairs, solved counts are reported but not tested, and Benjamini–Hochberg q < 0.05 is marked within a panel.Source: E. Scoring rules · p. 18
  • The scoring example separates solved-count change from speedup among kernels solved by both setups: the geometric mean rises from 2.44 to 2.97, with p = 0.36.Solved kernels rise from 7 to 8; the paired t-test interval is [0.75, 1.98].Source: E. Scoring rules · p. 18
  • Token accounting uses transcript tokenization when server counts are unavailable, and rebuilt prompts after compaction count as input.Prior work may sum every request’s prompt, making cached input a substantial and serving-engine-dependent part of multi-turn counts.Source: E. Scoring rules · p. 18
  • Weak scaling chooses decomposed sizes so work per PE stays fixed, with integer growth at P = m^k; MLScale grades from 1 to 16 GPUs across four nodes.MLScale reports the median over runs of the slowest GPU’s time and compares with PyTorch (torch.compile) on one GPU at base size.Source: E. Scoring rules · p. 18

F. Regenerating every figure, table and number

The release scripts pool final-grade answers, run paired tests, regenerate all figures and tables, and check 54 outputs against checksums.

F. Regenerating every figure, table and number

  • The reproduction workflow needs no GPU or model access and verifies all 54 regenerated outputs against SHA256SUMS.It uses shipped measurements; re-extraction checks every file against DATA_SHA256SUMS.Source: F. Regenerating every figure, table… · p. 19

G. Rerunning an experiment

The section documents the cluster setup, end-to-end experiment command, grading procedure, and workflow for rebuilding reported outputs. It also identifies paired intervals, rather than individual episodes, as the reproducibility target.

Platform

  • The experiments use MI300A nodes, with GH200 nodes for the regrade; CPU runs need CPU nodes, while GPU runs need AMD GPUs.The MI300A nodes have four APUs each; GH200 nodes have four superchips each, each with one H100.Source: G. Rerunning an experiment · p. 19

Grading

  • Grading jobs re-time answers on secret-seed inputs; unrun kernels are excluded, while kernels that ran and failed count as unsolved for success rate and token cost, but not ρS.Answers without a valid final measurement retain their last valid grade; answers without a final grade retain the judge's episode grade.Source: G. Rerunning an experiment · p. 19

One command per experiment

  • Each experiment command submits one Slurm job per setup, starting the model server, judge, and agents in fixed images; jobs finish within a 23-hour wall limit.SUBMIT=0 prints the plan without submitting, and Table XVII lists the commands, with a note to rerun RCCL-hint setups using PACKET=dist-rccl-amd.Source: G. Rerunning an experiment · p. 19, Table XVII · p. 20

One-time setup

  • Two cluster jobs build and device-check the judge, agent, and model-server images, and download model weights; the CPF setups also use pre-rendered forms.The images include ROCm 7.2.0; model serving uses SGLang for Qwen3.8-27B and Kimi-K2.7, and vLLM for OSS-120B.Source: G. Rerunning an experiment · p. 19

From finished runs to figures

  • The download workflow rebuilds data from run directories, and run_all.sh redraws figures, tables, and numbers; paired intervals, not individual episodes, are expected to reproduce.A checksum check names changed outputs; Table XVI lists scripts that run after stats.sh.Source: G. Rerunning an experiment · p. 20, Table XVI · p. 20

Related papers

Loading 2610.20673v1…