AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

Sharareh Younesian; Wenwen Ouyang; Sina Rafati; Mehdi Rezagholizadeh; Sharon Zhou; Ji Liu; Yue Liu; Yuchen Yang; Hao Li; Ziqiong Liu; Dong Li; Vikram Appia; Zhenyu Gu; Emad Barsoum

Source-linked AI summary

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

Sharareh Younesian, Wenwen Ouyang, Sina Rafati, Mehdi Rezagholizadeh, Sharon Zhou, Ji Liu, Yue Liu, Yuchen Yang, Hao Li, Ziqiong Liu, Dong Li, Vikram Appia, Zhenyu Gu, Emad Barsoum

arXiv:2605.16819v1May 16, 2026cs.CLcs.AIcs.LG

TL;DR

GPU kernel optimization needs specialized expertise, and existing benchmarks do not test complete agent workflows or transfer to unseen configurations. AgentKernelArena benchmarks these capabilities across 196 tasks with gated evaluation and centralized scoring, finding strong observed-input performance but a generalization weakness for kernels generated from scratch. Its scope is currently limited by hardware, agent, iteration, and API-cost constraints.

Problem
Existing kernel benchmarks do not evaluate full tool-using agent workflows, kernel-to-kernel optimization, and unseen-configuration generalization together.
Method
AgentKernelArena evaluates 196 HIP-to-HIP, Triton-to-Triton, and PyTorch-to-HIP tasks through isolated agent workflows, gated validation, centralized scoring, and unseen-configuration testing.
Results
Production agents achieve near-perfect compilation and high correctness on most categories, with speedups up to 6.89×, while PyTorch-to-HIP correctness drops by up to 40% on unseen configurations.
Takeaways & Limitations
HIP-to-HIP and Triton-to-Triton optimizations largely transfer to unseen shapes, whereas from-scratch PyTorch-to-HIP kernels often rely on shape-specific hardcoding.
Takeaways & Limitations
The study covers one GPU architecture, three commercial agents, three runs per configuration, and max iterations=3, with scope bounded by API cost.

Abstract

from arXiv · show

GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial low-level expertise. Recent AI coding agents can iteratively read code, invoke compilers and profilers, and refine implementations, yet existing kernel benchmarks evaluate single LLM calls rather than full agent workflows, and none include both kernel-to-kernel optimization and unseen-configuration generalization testing. We present AgentKernelArena, an open-source benchmark for measuring AI coding agents on GPU kernel optimization. The benchmark contains 196 tasks spanning HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation, and evaluates complete agent workflows in isolated workspaces using gated compilation, correctness, and performance checks, centralized scoring and an unseen-configuration generalization protocol that tests whether optimizations transfer to input configurations the agent never observed. Across production agents including Cursor Agent, Claude Code, and Codex Agent, we find near-perfect compilation and high correctness rates on most task categories, with the strongest configurations achieving mean speedups of up to 6.89x on PyTorch-to-HIP, 6.69x on HIP-to-HIP, and 2.13x on Triton-to-Triton tasks. Our unseen-configuration evaluation shows that HIP-to-HIP and Triton-to-Triton optimizations largely transfer to unseen input shapes, while PyTorch-to-HIP exhibits substantial correctness drops, indicating that agents generating kernels from scratch frequently hardcode shape-specific assumptions. AgentKernelArena is designed as a modular, extensible framework for rigorous evaluation of agentic GPU kernel optimization across agents, tasks, and hardware targets.

1 Introduction

GPU kernel optimization is important but traditionally demands deep hardware expertise, while existing benchmarks do not evaluate full tool-using agent workflows or unseen-configuration transfer. AgentKernelArena addresses these gaps with a modular benchmark spanning agent-centric evaluation and generalization testing.

GPU kernel optimization is a critical bottleneck because efficient implementation requires expertise in memory hierarchies, parallel execution, instructions, and architecture-specific hardware.
Existing benchmarks evaluate single-shot generation or light iterative prompting and omit both tool-using agent loops and kernel-to-kernel optimization.
AgentKernelArena contains 196 tasks across HIP-to-HIP, Triton-to-Triton, and PyTorch-to-HIP categories, evaluated in sandboxed workspaces through compile, correctness, and performance gates.
A centralized framework separates optimization from scoring to support fair and reproducible comparison across heterogeneous agent architectures.
The unseen-configuration protocol tests whether agent-optimized kernels transfer beyond observed inputs and exposes shape-specific assumptions that break on unseen configurations.
The modular design allows new agents, tasks, and hardware targets to be added through configuration.

2 Related Work

Prior benchmarks cover general code generation, repository-level agents, kernel generation, and specialized optimization systems, but use heterogeneous protocols or target domains distinct from GPU-kernel agentic optimization. AgentKernelArena provides a standardized domain-specific arena for comparing such systems.

HumanEval and MBPP assess short Python-function correctness, while SWE-bench and AgentBench evaluate repository-level or multi-environment agents in general software engineering.
KernelBench, TritonBench, ROCmBench, robust-kbench, and MultiKernelBench evaluate kernel generation across specifications, models, robustness checks, or hardware platforms.
LLM-driven systems such as QiMeng-Kernel, AutoTriton, TritonForge, AdaExplore, and GEAK use differing evaluation protocols, making cross-system comparison difficult.
Coding agents have shifted evaluation toward multi-turn, tool-augmented development, and AgentKernelArena applies this paradigm to performance-critical GPU kernel optimization.

3 AgentKernelArena: An Arena for Evaluating GPU Kernel Optimization Agents

AgentKernelArena evaluates autonomous GPU-kernel optimization in isolated workspaces across multiple task types, using gated validation, performance scoring, and unseen-configuration testing. Its design combines realistic task coverage with reproducible centralized evaluation.

Agent-centric evaluation: AgentKernelArena evaluates full agentic systems in siloed workspaces where agents compile, test, profile, and iteratively modify real GPU kernels.
Agent-centric evaluation: Workspace isolation prevents access to other tasks, prior runs, and other agents’ results, supporting reproducibility and parallel multi-GPU evaluation.
Execution flow: The pipeline baselines the original kernel, runs the agent, and centrally evaluates the modified kernel with compilation, correctness, and performance gates.
Task categories: The benchmark contains 196 tasks organized by source and target programming models, including HIP-to-HIP, Triton-to-Triton, and PyTorch-to-HIP.
Task categories: PyTorch-to-HIP requires creating an equivalent HIP kernel from scratch while handling memory layout, thread mapping, and numerical precision.
Generalization: Multiple visible input configurations encourage robustness across input geometries, unlike the separate protocol that evaluates configurations hidden from the agent.
Metrics: Compilation precedes correctness, and performance profiling occurs only after correctness passes, with speedup measured against task-specific baselines.
Metrics: The cumulative score assigns 20 points for compilation, 100 for correctness, and 100 · s_k for performance, with multi-shape speedup averaged arithmetically.

4 Experiments and Results

Across 196-task experiments, production agents generally compiled and preserved correctness while achieving category-dependent speedups. Unseen-configuration testing found strong transfer for HIP-to-HIP and Triton-to-Triton, but weaker correctness retention for PyTorch-to-HIP.

Main results: Agents achieved near-perfect compilation across categories, while HIP-to-HIP and Triton-to-Triton correctness rates were at least 91%.
Performance across categories: Mean speedups ranged from 3.74–6.89× for PyTorch-to-HIP, 1.44–6.69× for HIP-to-HIP, and 1.59–2.13× for Triton-to-Triton.PyTorch-to-HIP produced the largest gains, while Triton-to-Triton was the most challenging category.
Agent and model rankings: Claude Code with Opus 4.6 achieved the highest HIP-to-HIP mean speedup at 6.69×, while Cursor with Opus 4.7 High led Triton-to-Triton at 2.13×.
Unseen-configuration generalization: HIP-to-HIP conditional correctness ranged from 93.6% to 100%, and several configurations gained speedup on unseen inputs.Cursor with Opus 4.7 High achieved 100% conditional correctness with zero regressions.
Unseen-configuration generalization: Triton-to-Triton conditional correctness ranged from 90.9% to 99.4%, with generalization gaps below 0.1 for every configuration.
Unseen-configuration generalization: PyTorch-to-HIP had conditional correctness of 59.7% to 90.3% because generated kernels could hardcode dimension-specific assumptions that fail on unseen shapes.Correctly generalizing kernels nevertheless ran faster on unseen inputs.

5 Discussion

Agents use iterative, task-specific optimization strategies, with higher token budgets generally associated with better speedups. The study remains bounded by one GPU architecture, limited iterations and runs, selected commercial agents, and API-cost constraints.

Agent behavior patterns: Higher-capability agents use fusion, vectorized loads, warp-shuffle reductions, and launch-bound tuning on HIP-to-HIP tasks.
Agent behavior patterns: Triton-to-Triton optimization centers on autotuning block size, warps, stages, and AMD-specific waves-per-EU settings.
Agent behavior patterns: PyTorch-to-HIP tasks require agents to map high-level module semantics onto threads, blocks, grids, memory allocation, and Python bindings.
Computational cost: 39–86K output tokens per task make Claude Code the most verbose agent, while higher token budgets generally correlate sublinearly with speedups.
Scope and limitations: The study covers one GPU architecture, three commercial agents, max iterations=3, and three runs per configuration, with broader hardware and iteration sensitivity left for future work.
Broader impact: The framework supports adding agents, tasks, and hardware targets through launcher scripts, YAML configuration, task directories, and architecture cheatsheets.

6 Conclusion

AgentKernelArena is an open-source benchmark for evaluating AI coding agents on GPU kernel optimization across three task categories. Its gated evaluation and unseen-configuration protocol show strong visible-configuration performance but expose shape-specific failures in kernels generated from scratch.

AgentKernelArena evaluates 196 HIP-to-HIP, Triton-to-Triton, and PyTorch-to-HIP tasks through gated compilation, correctness, and performance checks with centralized scoring.
Up to 6.89× speedup over baseline is reported, while PyTorch-to-HIP correctness drops by up to 40% on unseen input configurations.
Unseen-input evaluation reveals shape-specific hardcoding in kernels generated from scratch and shows why visible-configuration performance alone can overestimate reliability.

A Task Curation Process

AgentKernelArena curates 196 self-contained GPU kernel tasks from multiple sources and validates them with standardized tooling. Its configuration-driven design gives agents identical prompts and sandboxes while centralizing evaluation and enabling extensibility.

A Task Curation Process: 196 tasks are curated from HIP-to-HIP, PyTorch-to-HIP, Triton-to-Triton vLLM, and Triton-to-Triton ROCmBench sources.
Task validation: All 196 tasks undergo automated validation of directory structure, compilation, correctness, and performance, with failed tasks fixed or excluded.
Task format: Each task directory contains source code, evaluation scripts, and config.yaml controlling prompt generation and scoring.
Agent evaluation setup: Agents receive seven-section prompts and operate with identical prompts, sandboxed workspaces, full shell access, and up to three successive kernel versions.
Agent evaluation setup: Each agent, model, and task-type configuration is run three times, with per-task metrics averaged across runs before aggregate statistics are computed.
Reproducibility: Model identifiers and calendar windows are reported because stable provider identifiers do not fully pin model versions over time.
Extensibility: New agents require a registered launcher and agent config, after which the centralized evaluator handles scoring without pipeline changes.
Extensibility: New tasks require configuration, kernel and reference source files, and task-specific compilation, correctness, and performance scripts.

G Speedup Distribution Details

The benchmark reports cross-task speedup distributions alongside mean and geometric mean results. These distributions distinguish variation across kernels from run-to-run variability and reveal substantial heterogeneity in Triton performance.

Speedup distributions: Table 8 reports standard deviation, P25, P75, and P90 for each configuration alongside mean and geometric mean speedups.
Speedup distributions: 6–9× cross-task standard deviations can accompany Triton-to-Triton mean speedups near 2× because a few tasks have exceptional optimization headroom.

H Agent Behavior Analysis

The analysis tracks complete agent workflows across task categories, showing universal task completion but task-specific failure patterns and optimization strategies.

Iteration and completion: All agents complete every HIP-to-HIP and PyTorch-to-HIP task, while all evaluated agents complete the full Triton-to-Triton set.The categories contain 24, 24, and 148 tasks respectively.
Compilation and correctness failures: PyTorch-to-HIP produces the most compilation churn, whereas Triton failures are mainly autotuning-related correctness errors.Claude Code encounters approximately 35 compilation failures across 24 PyTorch-to-HIP tasks; Triton errors include misconfigured autotuning parameters.
Optimization strategies by task type: HIP-to-HIP agents emphasize coalescing, vectorization, fusion, launch bounds, and shared-memory management.These strategies target memory access, computation, and AMD CDNA3-specific features.
Optimization strategies by task type: Triton-to-Triton agents primarily tune BLOCK SIZE, num warps, num stages, waves per eu, and matrix instr nonkdim.The passage identifies autotuning configuration as the dominant Triton optimization focus.
Optimization strategies by task type: PyTorch-to-HIP agents additionally map PyTorch module semantics to HIP kernel launches.This category requires solving translation and launch-mapping challenges beyond kernel optimization alone.

I Failure Case Analysis

Failure analysis shows that centralized and unseen-configuration checks expose compilation, numerical, type, autotuning, and shape-specific errors that original tests can miss.

PyTorch-to-HIP compilation failures: PyTorch-to-HIP compilation failures arise mainly from malformed PYBIND11 MODULE entry points that omit the required PyInit symbol.The failure occurs because agents generate complete HIP kernels with Python bindings from scratch.
PyTorch-to-HIP compilation failures: Approximately 70% compilation is achieved by Cursor Agent with GPT-5.4 High, while other listed configurations reach 95–100% on the same PyTorch-to-HIP tasks.GPT-5.4 High fails on 7 of 24 tasks per run; the passage links this variation to wiring torch.utils.cpp extension.load() bindings.
HIP-to-HIP correctness failures: HIP-to-HIP correctness reaches at least 91.7%, with rare tolerance violations reflecting numerically acceptable but non-bitwise-identical outputs.The passage characterizes modifying existing working code as more forgiving than generating kernels from scratch.
Triton-to-Triton correctness failures: Triton correctness is 96–100%, with failures from conditional-branch type mismatches and autotuning-induced numerical errors.Specific issues include JIT failures for parameter combinations and FP8 rounding differences on the moe gemm task.
Unseen-configuration failures: 54–92% unseen-input retention occurs for PyTorch-to-HIP, where hardcoded buffer sizes, block dimensions, and loop bounds break on unseen shapes.Examples include MAX ROWS=32 and MAX C=32 assumptions causing runtime failures despite 100% correctness on original shapes.
Unseen-configuration failures: Cursor Agent with GPT-5.4 High regresses on 8–11 of 24 unseen-shape tasks, while Cursor Agent with Opus 4.6 High reaches 92% retention.HIP-to-HIP retains 91–100%, and Triton-to-Triton retains 90–100%, with Triton failures including power-of-two constraints and shape-dependent numerical error.
Implications for benchmark design: Centralized evaluation, multi-shape and multi-dtype tests, and unseen-configuration testing expose failures that limited agent-side testing can miss.These design choices also reveal hardcoded assumptions that inflate reported speedups.

K Example Agent Interaction

The example session illustrates an iterative compile–test–profile workflow in which the agent improves performance, diagnoses correctness failures, and adapts after resource-limit errors.

Initial workflow: The agent reads the Triton source, identifies fused_moe_kernel, and verifies compilation before running correctness and performance tests.The initial workflow uses task_runner.py for compilation, correctness, and performance checks.
Optimization and correction: A tl.dot accumulator optimization compiles successfully, but a BLOCK_SIZE_K change later fails correctness and is reverted with masking adjustments.The failed test exceeds absolute tolerance on test case 3.
Performance feedback: Performance tests report 12.9–18.1% reductions in example runtimes after optimization.The reported values are shown for test cases 1 and 2.
Resource-limit recovery: A shared-memory prefetching attempt exceeds the LDS limit, after which the agent reduces tile size and obtains a further 5% improvement.Compilation succeeds after the tile-size adjustment and correctness passes.
Final evaluation: The centralized evaluator reports a final average speedup of 1.23x across five test shapes.This final rerun includes compilation, correctness, and performance checks.

L Limitations

The study is scoped to AMD MI300X and a limited set of commercial-agent evaluations, while its prompt assembly includes an agent-launcher iteration directive.

Scope and evaluation limits: The study targets a single GPU architecture, AMD MI300X, and evaluates three commercial agents over three runs per configuration because of API-cost limits.Model availability varies across platforms, so most comparisons use Cursor Agent.
Scope and evaluation limits: Open-weight models were explored with single-iteration calls but consistently failed compilation on the large multi-file contexts, outside the benchmark’s iterative-feedback scope.The study therefore does not provide comparable iterative results for those models.
Scope and evaluation limits: Specialized systems such as GEAK and AutoTriton were excluded because their task-specific architectures do not permit a fair comparison with general-purpose coding agents.The framework is described as supporting their future integration.
Scope and evaluation limits: The task set is drawn primarily from vLLM and GPU Mode and is being expanded with kernels from repositories such as AITER.The dataset is therefore not presented as covering all kernel repositories.
Prompt configuration: The fused MoE prompt contains eight sections and may append a natural-language iteration directive when the launcher sets a maximum iteration count.The default maximum is three iterations.