KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

• KernelGenBench introduces a unified benchmark spanning diverse operator sources and six hardware platforms through a common Triton target.Its controlled views evaluate 210 operators across PyTorch ATen, vLLM, and cuBLAS, plus a 110-operator subset across six chips.Source: §1 · p. 2
• Agent-based methods outperform pure LLM sampling, but no method dominates across sources and platforms.ATen is most tractable, vLLM is hardest to generate correctly, and cuBLAS provides the strongest performance-ceiling stress test.Source: §1 · p. 3
• 15 billion tokens were consumed to evaluate correctness, speedup, and agentic cost efficiency at scale.The protocol combines distributed sandboxing with layered anti-hack checks and multidimensional metrics.Source: §1 · p. 3
• AutoKernel accuracy falls from 87% on NVIDIA to 25% on Iluvatar CoreX, showing substantial cross-platform variation.Source: §1 · p. 3
• Specialized agents average 4.99 million tokens per successful operator, rising to 6.25 million for CUDA Optimized Skill.Source: §1 · p. 3

• Existing benchmarks diversify tasks, architectures, production contexts, and agentic workflows, but use varied evaluation settings.Source: §2.2 · p. 3
• KernelGenBench instead compares ATen, vLLM, and cuBLAS under a fixed API-replacement protocol and adds layered anti-hack enforcement and agentic cost.Source: §2.2 · p. 3
• Most prior evaluation remains NVIDIA-centric or architecture-specific, while multi-platform benchmarks often use separate platform-specific languages.Source: §2.2 · p. 3

KernelGenBench-MS

• Multi-source evaluation separates framework-semantic reproduction, specialized production-operator reconstruction, and performance against expert-optimized libraries.Source: §3.1 · p. 4
• The MS suite contains 110 PyTorch ATen, 50 vLLM, and 50 cuBLAS problems with source-specific specifications and references.Source: §3.1 · p. 4

Controlled views

• KernelGenBench-MS varies source across 210 NVIDIA operators, while KernelGenBench-MC varies six hardware stacks on a semantically stable 110-operator ATen suite.Source: §3.2 · pp. 4–5, §3.1 · p. 4

KernelGenBench-MC

• A common Triton target enables evaluation across six platforms, with hardware detection, platform constraints, adaptive tolerances, and anti-hack configurations.Source: §3.2 · p. 5

Kernel generation

• The evaluation compares zero-shot and multi-sample LLM generation with execution-feedback agents under a shared 30-minute task budget.Source: §3.3 · p. 5

Verification

• Combinatorial tests vary semantic parameters, shapes, data types, and memory layouts rather than relying on a single fixed shape.Source: §3.3 · p. 5

Anti-hack architecture

• Three anti-hack tiers scan syntax, replay execution with the Triton function disabled, and profile hardware activity.These checks intercepted 13 Pass@1 and 30 Pass@5 shortcut submissions during evaluation.Source: §3.3 · p. 5

Evaluation metrics

• Correctness requires all test cases and available anti-hack checks to pass, while performance uses a two-level geometric mean across tests and solved operators.Source: §3.3 · p. 5

4.2 KernelGenBench-MS: Multi-Source Evaluation

• 87% accuracy was achieved by Claude Code, while AutoKernel reached the highest successful-set speedup at 1.02×, but these metrics use different successful sets and do not establish an overall winner.The 4 pp accuracy gap between Claude Code and AKO4all is descriptive under single-trajectory evaluation, not statistically established.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 KernelGenBench-MC: Cross-Platform Evaluation

• AutoKernel accuracy fell from 87% on NVIDIA to 25% on Iluvatar CoreX, showing that a common Triton target does not ensure generation-system portability.Claude Code remained relatively accurate across platforms, yet Platform A reduced its speedup from 0.86× to 0.18×; Iluvatar also changed relative system ordering, not only absolute success rates.Source: §4.3 · p. 7, Table 2 · p. 7
• Non-NVIDIA platforms required 1.27–2.06× the tokens and 1.67–2.00× the time of NVIDIA, with Platform A reaching 173M tokens and 18 hours versus 84M and 9 hours.The portability tax is defined as additional token or wall-clock cost for a fixed 110-operator task set, model, and scaffold relative to NVIDIA.Source: §4.3 · p. 7, Figure 2 · p. 8

4.4 Source-Conditioned Capability Profiles

• vLLM was the strongest correctness challenge but offered greater optimization headroom, whereas cuBLAS allowed high correctness while imposing a stronger vendor-tuned performance ceiling.vLLM median accuracy was 37 pp below ATen and exceeded ATen speedup in 15 of 20 defined pairs; cuBLAS speedup was below ATen in 18 of 20.Source: §4.2 · p. 6, §4.4 · p. 8, Table 3 · p. 8

4.5 Why Portability Fails: Trajectory Analysis

• Trajectory auditing attributed portability failures partly to platform interfaces, including recurring type rejection on Moore Threads and API, language-feature, dtype, compiler, and runtime problems on Iluvatar CoreX.At least 10 of 110 Iluvatar operators were affected by explicit API, language-feature, or dtype incompatibilities.Source: §4.5 · pp. 8–9

4.6 Agentic Scaffolds: Gains, Costs, and Search Behavior

• Specialized agents averaged 4.99M tokens per successful operator, while CUDA Optimized Skill reached 6.25M and AKO4all consumed 904M tokens, making accuracy and speedup alone insufficient for operational evaluation.Interactive debugging produced 87% accuracy, 27 pp above fixed traceback-driven reflection, but fixed reflection underperformed independent sampling on vLLM and showed higher cross-turn code similarity on jointly failed operators.Source: §4.6 · p. 9, Table 4 · p. 9

5 Conclusion

• KernelGenBench evaluates 210 operators from three sources on NVIDIA and a semantically stable 110-operator set across six hardware platforms.The benchmark uses a common Triton target and supports broader cross-vendor infrastructure beyond the two reported views.Source: §5 · p. 9
• Operator source, hardware platform, and agentic scaffold emerge as distinct capability dimensions, with source changing the correctness and optimization challenge.The conclusion also reports poor cross-platform transfer and higher correctness from more interactive scaffolds at high cost.Source: §5 · p. 9
• Public-reference mappings are not uniformly semantically matched, costly configurations mostly use single trajectories, and two platforms remain anonymous.The authors identify broader aligned references, repeated evaluation under practical budgets, and additional vendor access as future work.Source: §5 · p. 9

Prompt Design

• KernelGenBench’s prompt system dynamically assembles structured, source-specific context while keeping production kernel implementations hidden from models.The templates distinguish interface-level wrapper or harness information from the underlying implementation.Source: A Prompt Design · p. 14, Table 6 · p. 14

Evaluation Infrastructure

• The infrastructure supports hardware-aware templates, full dispatch registration, isolated parallel execution, anti-hack checks, and platform-local reference verification.These mechanisms evaluate generated kernels through platform-specific runtime paths rather than isolated function calls.Source: D Cross-Platform Prompt and Tolerance… · p. 18, E Distributed Sandbox Infrastructure · pp. 18–19

Evaluation Setup

• All methods share operator specifications, test suites, anti-hack checks, and a 30-minute task budget, while retaining method-specific tools and reporting token and wall-clock costs.The benchmark controls external task conditions rather than internal agent iterations.Source: F Evaluation Pipeline Algorithm Details · p. 19

Agentic Evaluation

• Agentic dispatch builds a device-specific prompt, launches an isolated process, and lets the agent write, verify, debug, and optimize until completion or timeout.Operators run in parallel subject to available-device bounds.Source: F Evaluation Pipeline Algorithm Details · p. 19

LLM Sampling

• LLM Sampling draws fresh independent samples without prior candidates or execution feedback, stops sampling an operator after correctness, and applies anti-hack checks afterward.Correctly solved operators are removed from later rounds through correctness-based early stopping.Source: F Evaluation Pipeline Algorithm Details · p. 19, G Stress Test Shape and Dispatch… · p. 20

Test Construction

• Test construction stresses non-aligned long-tail sizes, scalar-to-5D tensors, minimal sizes, and operator-specific shape, dtype, axis, and stride combinations.For softmax, all 18 Cartesian-product cases must pass adaptive numerical tolerance.Source: G Stress Test Shape and Dispatch… · pp. 19–20, Table 13 · p. 20

Generation Bottlenecks

• GLM-5.0 frequently truncates kernels under the uniform 16K-token limit because verbose intermediate variables and comments consume generation space.The resulting incomplete core logic contributes to functionally incorrect Pass@5 submissions.Source: H Additional Analysis on Generation… · p. 20

Anti-Hack Analysis

• Anti-hack checks reject both direct reuse of existing ATen operators and hybrid implementations that delegate required reductions to PyTorch.The layered checks combine explicit-call detection with ghost replay and reject partial high-level fallbacks.Source: H Additional Analysis on Generation… · pp. 20–21, Table 15 · p. 21
• 29 of 30 Pass@5 anti-hack interceptions occur in ATen, while vLLM accounts for one, consistent with ATen exposing more easily callable high-level operators.These interceptions are excluded from clean pass rates.Source: H Additional Analysis on Generation… · p. 21

Geometric Mean

• The two-level geometric mean aggregates test-case speedups within operators and then aggregates operator speedups across the successfully solved set.All main tables report GeoMean, while global speedup remains a successful-set statistic rather than a matched-set ranking.Source: I Speedup Statistical Metrics · pp. 21–22, §2 · p. 22
• Geometric averaging treats reciprocal slowdowns and speedups symmetrically, unlike arithmetic averaging, and prevents operators with more test cases from receiving disproportionate weight.A 0.5× and 2.0× pair yields a neutral 1.0× geometric mean rather than an overstated 1.25× arithmetic mean.Source: I Speedup Statistical Metrics · p. 22

Metric Roles

• GeoMean, Median, and IQM serve complementary roles for multiplicative aggregation, conservative estimation, and tail-robust averaging.The framework uses GeoMean as the primary metric and the other two as auxiliary summaries.Source: I Speedup Statistical Metrics · p. 22, §2 · p. 22

Median

• Median provides a conservative central estimate that is resistant to extreme outliers when inter-operator speedup variance is large.The metric sorts operator-level speedups and selects the middle value or the average of the two middle values.Source: §1 · p. 22

Interquartile Mean

• IQM averages the central half of the operator-speedup distribution after trimming 25% from each tail.It balances mean efficiency with median-like robustness and handles fractional boundary observations.Source: §2 · p. 22

Repeated-run stability

• 10–12% run-to-run variation was observed for one five-run, 210-operator sampling configuration, providing a protocol-level stability check rather than main-table variance estimates.The repeated model and decoding configuration differed from the main table, so closely spaced single-trajectory results remain descriptive.Source: J Repeated-Run Robustness Analysis · p. 23, Table 17 · p. 23

Repeated-run coverage

• Most agent results are single trajectories because repeating every model–agent–source–platform configuration would multiply the study’s 15-billion-token, approximately $20,000 evaluation cost.Aggregating 110 or 210 task outcomes reduces individual-task influence but does not replace configuration-level repeats.Source: K Additional Reproducibility Details… · p. 23

Protocol boundaries

• The benchmark fixes prompts and task protocols without per-operator tuning, enabling future prompt-sensitivity studies under the same verification pipeline.Source: K Additional Reproducibility Details… · p. 23

Verification coverage

• L3 profiling is NVIDIA-only because a portable profiler interface is unavailable across vendor stacks, while L1 static scanning and L2 ghost replay are shared across platforms.Representative shortcut behaviors are already rejected by L1/L2 verification.Source: K Additional Reproducibility Details… · p. 23

Metric interpretation

• Global speedup is computed over each method’s successful operator set, so matched-intersection analysis is required for identical-set performance comparisons.The source references also differ in abstraction and optimization level, motivating emphasis on per-source accuracy and speedup.Source: K Additional Reproducibility Details… · p. 23

System-level evaluation

• The benchmark compares complete LLM–agent systems while preserving their prompts, skills, tools, and native profiling access, rather than isolating language models.Source: K Additional Reproducibility Details… · p. 23

Operator difficulty

• The difficulty score averages whether four general-agent configurations solve each operator, while matched-set analysis uses clean post-verification successes.Source: N Operator Difficulty and Matched-Set… · p. 24

Evaluation scope

• The benchmark reports strict speedup compliance across 210 NVIDIA operators and 110 ATen operators on six heterogeneous platforms, alongside agentic token and execution costs.Source: Table 20 · p. 25, Table 21 · p. 25, Table 22 · p. 25

Operator difficulty

• Table 23 counts how many of four configurations solve each operator, exposing consistency patterns rather than treating aggregate accuracy as uniform difficulty.Source: Table 23 · p. 26
• 114/210 operators are solved by all four agents, whereas 20/210 are never solved, separating broadly stable difficulty from configuration-sensitive difficulty.vLLM contributes 13 of the 20 stable-hard tasks, while 75/110 ATen tasks are solved universally.Source: N Operator Difficulty and Matched-Set… · pp. 24, 26

Matched-set results

• For Opus-4.6, every matched-set interval includes 1, while GLM-5.0 reverses the faster scaffold by source: CC on shared ATen successes and OC on shared vLLM successes.The overall GLM interval also includes 1, and the small vLLM intersection limits interpretation to source-dependent scaffold interaction.Source: O Accuracy–Speedup Gap · p. 26, Table 24 · p. 26

Accuracy–speedup relationship

• Accuracy spans 2–87% while successful-set speedup ranges from 0.62× to 1.01×, with 12 of 16 configurations between 0.68× and 0.83×.The figure covers LLM-sampling and vanilla-agent configurations on NVIDIA.Source: O Accuracy–Speedup Gap · p. 26, Figure 5 · p. 27

Matched-set comparison

• A high successful-set GeoMean need not indicate stronger optimization because failures remove difficult operators, so matched-set comparisons evaluate methods on common successful operators.Source: O Accuracy–Speedup Gap · p. 26

Failure taxonomy

• Model-code failures include infinite dispatch recursion, hallucinated Triton APIs, and omitted overload variants, while platform failures reflect compiler, runtime, or backend limitations.The benchmark distinguishes these categories so non-model infrastructure errors are not counted as algorithmic model errors.Source: P Platform-Specific Failure Patterns · p. 27

Platform-specific failures

• Iluvat​​ar CoreX exhibits at least 10 explicit incompatibilities across 110 operators, including unavailable math APIs, unsupported language features, and incomplete complex or BFloat16 support.This conservative mechanism-specific lower bound covers 9.1% of the suite; compiler hangs and crashes are additional evidence.Source: P Platform-Specific Failure Patterns · p. 27
• Moore Threads exposes stricter type inference, Hygon exposes backend-specific Triton semantics, and Iluvatar CoreX exposes API and dtype coverage gaps.Platform A combines compilation and runtime instability, whereas Platform B exposes IR-pass and addressing constraints.Source: P Platform-Specific Failure Patterns · p. 27, Table 25 · p. 28

Ablation breakdown

• Figure 6 reports accuracy bars and speedup lines across Overall, ATen, vLLM, and cuBLAS, with interactive debugging showing its largest gain on vLLM at +44 percentage points.Source: Figure 6 · p. 29, Q Generation-Regime Comparison:… · p. 29

Reflection analysis

• For 21 jointly failed operators, Reflection clusters cross-round code similarity through local patching, whereas No-Reflection spreads similarity through broader search.Source: R Reflection Mode Per-Operator Code… · p. 29, Figure 7 · p. 30, Table 27 · p. 30