PaperlayerView exact revision on arXiv

Source-linked AI summary

KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu

arXiv:2605.04956v2cs.LGcs.PF

TL;DR

It remains unclear where LLM-generated Triton kernels fail and how iterative refinement affects correctness and efficiency. KernelBenchX evaluates these questions across task categories, correctness, and hardware efficiency, revealing distinct barriers from compilability through performance portability.

  • Problem

    Existing benchmarks provide limited evidence about which kernel-generation tasks succeed or fail, how refinement affects outcomes, and why.

  • Method

    KernelBenchX evaluates Triton kernels with a two-stage correctness protocol, a 15-category taxonomy across 176 tasks, and hardware-efficiency metrics.

  • Results

    Kernel-generation capability is structured by task category, while iterative refinement expands feasible solutions but performance validity remains a distinct challenge.

  • Takeaways & Limitations

    Progress requires reasoning about global tensor contracts and reduction semantics, numerical fidelity, and hardware costs rather than relying on prompt engineering or refinement alone.

  • Takeaways & Limitations

    FLOP- and byte-based metrics are normalized efficiency proxies derived from fixed task-level target models, not measurements of actual executed instructions.

Abstract

from arXiv · show

LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We present KernelBenchX, a benchmark designed to answer this question through category-aware evaluation of correctness and hardware efficiency across 176 tasks in 15 categories. Our systematic comparison of five representative methods yields three main findings. First, task structure determines correctness more than method design. Category explains nearly three times more variance in semantic correctness than method (9.4% vs 3.3% explained deviance), and 72% of Fusion tasks fail across all five methods while Math tasks are solved consistently. Second, iterative refinement improves correctness, but not performance. Across GEAK iterations, compile rate rises from 52.3% to 68.8% while average speedup declines from $1.58\times$ to $1.44\times$; newly rescued kernels consistently underperform persistently correct ones ($1.16\times$ vs $1.58\times$ speedup in round~0$\to$1). Third, correctness does not imply efficiency. 46.6% of correct kernels are slower than the PyTorch eager baseline, and cross-hardware speedup variance reaches $21.4\times$. Besides, quantization remains completely unsolved (0/30 successes) despite non-trivial compilation rates, revealing systematic misunderstanding of numerical computation contracts rather than surface-level syntax errors. These findings suggest that future progress depends on handling global coordination, explicitly modeling numerical precision, and incorporating hardware efficiency into generation. The code is available at https://github.com/BonnieW05/KernelBenchX

1 Introduction

KernelBenchX addresses unresolved questions about the capability boundaries and effects of iterative refinement in LLM-based Triton kernel generation. It combines category-aware correctness analysis with hardware-efficiency evaluation in a unified benchmark and experimental pipeline.

  • Limitations: Existing benchmarks do not adequately explain which task types current kernel-generation methods handle reliably, which fail, or how iterative refinement affects compilation, correctness, and performance.These limitations motivate a benchmark focused on capability boundaries and refinement behavior.
  • Our Method: KernelBenchX extends TritonBench-T with a two-stage correctness protocol, a unified 15-category taxonomy including quantization and multi-precision tasks, and hardware-efficiency metrics beyond runtime.The correctness protocol is designed to reject implementations that pass output comparison by chance.
  • Experimental Overview: The study systematically compares representative Triton kernel-generation methods across task categories, correctness outcomes, and efficiency metrics under one evaluation pipeline.The evaluation also collects error-correction and optimization pairs for future training and inference-time improvement.
  • Contributions: KernelBenchX provides category-aware evaluation of Triton kernel-generation correctness and hardware efficiency.This benchmark is the paper’s central contribution.
  • Contributions: The paper identifies three empirical findings that characterize LLM-based kernel-generation capability boundaries and supplies mechanistic analysis for each.The findings are presented as the central empirical contribution.
  • Contributions: The released error–correction and optimization pairs support future training and inference for kernel generation.These pairs are collected during evaluation.

2 Preliminary

This section defines hardware-efficiency metrics by normalizing achieved bandwidth and throughput against hardware peaks. It also describes five kernel-generation methods spanning general-purpose and specialized models, iterative refinement, and domain-specific training.

  • Hardware efficiency: Hardware efficiency evaluates resource use beyond runtime through achieved bandwidth and achieved throughput.For kernel k, runtime, bytes moved, and floating-point operations define these measurements.
  • Hardware efficiency: Peak-normalized bandwidth and throughput enable hardware-independent comparison.The quantities are reported in GB/s and TFLOPS before normalization by corresponding peak values.
  • Hardware efficiency: IOU measures memory-bandwidth utilization, whereas MFU measures compute utilization.The metrics are complementary because kernels may be memory-bound or compute-bound.
  • Compared methods: Five methods span general-purpose versus specialized models, iterative refinement, and domain-specific training.The comparison includes AutoTriton, GEAK, KernelAgent, Claude, and DeepSeek-Coder.
  • Compared methods: GEAK uses generator, evaluator, reflector, and optimizer modules with three iterations and four candidates per round.It uses DeepSeek-V3.2-Chat, temperature 1.0, and retains five best implementations as context per round.
  • Compared methods: KernelAgent uses generate–verify–refine with three parallel workers and up to five refinement rounds.It uses DeepSeek-V3.2-Chat at temperature 0.4, bypassing the Fuser pipeline for single-operator tasks.

3 Methodology

KernelBenchX evaluates Triton kernel generation across 176 structurally categorized tasks using shared specifications, method adapters, and a unified pipeline. Its protocol combines compilation, constraint checks, distribution-shifted execution accuracy, runtime benchmarking, and normalized hardware-efficiency analysis.

  • Evaluation pipeline: A shared task specification and unified evaluation pipeline generate and assess candidates from multiple methods for correctness, efficiency, and code quality.Each task provides a function interface, reference implementation, and task-specific constraints; intermediate logs support error analysis.
  • Benchmark coverage: 176 tasks span 15 computationally structured categories, enabling comparisons among tasks with similar parallel execution requirements.Categories include Activation, Convolution, Fusion, Index, LinearAlgebra, Loss, Math, MatrixMultiply, Normalization, Optimizer, Pooling, Quantization, Random, Reduce, and SpatialOps.
  • Benchmark coverage: Multi-precision coverage tests fp16, bf16, and int8 variants, while six W8A8 and W4A16 tasks require manual quantization logic.Quantization tasks test scale computation, explicit casting, and dequantization without high-level APIs.
  • Correctness protocol: The two-stage correctness protocol combines call validation with execution checks across standard and outlier input distributions.Outlier mode uses probability 0.1% and scale factor 50, while quantization tasks additionally undergo static API and manual-logic checks.
  • Efficiency and quality metrics: Runtime uses 25 warmup and 100 measurement runs with median reporting, while speedup compares against PyTorch eager and efficiency uses max(IOU, MFU).Code quality is measured with Maintainability Index and Cyclomatic Complexity; FLOP- and byte-based quantities are normalized proxies from a fixed task-level target model.

4 Experiments

Experiments across five methods show that compile success, semantic correctness, and useful acceleration are distinct outcomes. Task category dominates correctness, iterative refinement improves correctness while reducing speed, and many correct kernels remain inefficient or hardware-sensitive.

  • Aggregate outcomes: 64.2% of KernelAgent kernels compile, but only 10.8% are correct, while GEAK achieves the highest overall correctness at 30.7%.Correct/Compile conversion for KernelAgent is 16.8%; Claude reaches 22.7% overall correctness.
  • Category effects: Quantization and SpatialOps convert compilable candidates to correct kernels at 0%, while Activation and Math reach 46–56%.Fusion and MatrixMultiply remain near 25%, despite non-trivial compile rates in hard categories.
  • Iterative refinement: Compile success rises from 52.3% to 68.8% and correctness from 18.2% to 30.7%, while average speedup falls from 1.58× to 1.44×.Newly rescued kernels in rounds 0→1 average 1.16× speedup versus 1.58× for already-correct kernels.
  • Efficiency and portability: 46.6% of correct kernels are slower than eager PyTorch, the pooled median speedup is 1.0008×, and worst-case cross-machine speedup variation reaches 21.4×.The fraction slower than PyTorch ranges from 18% on A100 to 76% on L20.

5 Insights

KernelBenchX identifies three distinct bottlenecks: task structure drives semantic correctness, iterative refinement mainly repairs compilability without reliably improving performance, and efficiency remains unresolved even for correct kernels. Hardware-dependent speedups further show that correctness and performance are separate frontiers.

  • Global-Contract Semantic Failure: 9.4% vs. 3.3%: task category explains nearly three times more variance in semantic correctness than method identity.Correctness gaps persist despite non-trivial compile rates, while static complexity proxies correlate only weakly with failure (r ≤0.21).
  • Repair-Biased Iterative Refinement: Iterative refinement expands compilability and correctness, but kernel performance often fails to improve and can degrade across iterations.The 352 GEAK diffs indicate that current loops primarily repair explicit local errors, while performance-oriented rewrites are rare.
  • Performance as an Unsolved Frontier: 46.6% of semantically correct kernels remain slower than eager PyTorch, with a pooled median speedup of only 1.0008×.This indicates that semantic correctness alone does not establish practical efficiency.
  • Performance as an Unsolved Frontier: 21.4×: the worst-case max/min speedup ratio across machines shows that correct kernels are often hardware-specific rather than generally efficient.The findings distinguish correctness from performance and motivate mechanisms such as explicit hardware-aware search or performance optimization.

6 Conclusion

KernelBenchX characterizes LLM-based Triton kernel generation across 176 tasks on six GPUs, revealing distinct barriers in compilability, semantic correctness, hardware efficiency, and performance portability. The findings show that iterative refinement expands feasibility but remains repair-biased, while prompt engineering and refinement are insufficient beyond compilability.

  • Conclusion: KernelBenchX evaluates LLM-based Triton kernel generation across 176 tasks on six GPUs using category-aware evaluation.The benchmark is introduced to characterize the capability boundary of LLM-based Triton kernel generation.
  • Conclusion: Iterative refinement expands the feasible set but introduces weaker candidates, with edits dominated by local fixes.The conclusion characterizes refinement as repair-biased and states that performance validity remains unsolved.
  • Conclusion: The capability boundary comprises distinct barriers: compilability, semantic correctness, hardware efficiency, and performance portability.Each barrier requires different mechanisms to clear, rather than a single mechanism addressing one overall failure boundary.
  • Conclusion: Prompt engineering and iterative refinement suit compilability but are structurally insufficient for semantic correctness, hardware efficiency, and performance portability.The conclusion presents these methods as inadequate for clearing the remaining barriers.

Technical Appendices and Supplementary Material · A Benchmark and Evaluation Details

The supplementary material organizes the benchmark and evaluation details alongside detailed results, quantization analysis, explanations of kernel-generation failures, and repair or optimization artifacts.

  • A Benchmark and Evaluation Details: Section A presents the benchmark and evaluation details.
  • Technical Appendices and Supplementary Material: Section B contains detailed results.
  • Technical Appendices and Supplementary Material: Section C provides quantization details.
  • Technical Appendices and Supplementary Material: Section D analyzes why LLMs cannot reliably generate high-performance kernels.
  • Technical Appendices and Supplementary Material: Section E documents transition pairs for repair and optimization analysis.
  • Technical Appendices and Supplementary Material: Together, Sections A–E define the supplementary benchmark, results, quantization, failure-analysis, and artifact components.

A.1 Task List and Category Taxonomy … B Detailed Results

KernelBenchX organizes 176 tasks across 15 knowledge-based categories and evaluates kernel validity, execution correctness, and performance using explicit benchmark criteria. Its taxonomy spans specification translation, parallel aggregation, structured computation, spatial indexing, fusion, and semantic contracts.

  • A.1 Task List and Category Taxonomy: 176 tasks span 15 fine-grained categories grouped by the knowledge required for correct implementation rather than operator type.This organization supports category-level analysis of systematic failure modes.
  • A.1 Task List and Category Taxonomy: Activation (10) and Math (36) primarily require faithful translation of explicit formulas or standard library semantics.Composition and numerical stability can still introduce subtleties.
  • A.1 Task List and Category Taxonomy: Reduce (6), Pooling (2), and Normalization (5) depend on aggregation over defined scopes, with errors arising from incorrect scope or inconsistent statistics.Some variants extend to matrix-level aggregation, including spectral normalization.
  • A.1 Task List and Category Taxonomy: MatrixMultiply (10) and LinearAlgebra (17) require structured multi-operand computation and consistency across coupled outputs.LinearAlgebra examples include SVD, QR, and LU decompositions with constraints such as A = USV H.
  • A.1 Task List and Category Taxonomy: Fusion (60) composes multiple operations in one kernel, making correctness depend on preserving invariants across operation boundaries.These boundaries may combine elementwise, reduction, and normalization steps.
  • A.1 Task List and Category Taxonomy: Loss, Optimizer, Random, and Quantization require semantic contracts beyond direct formulas, including API behavior, seeding, device semantics, and approximation metrics.Quantization is evaluated by multiple precision metrics rather than exact equality.
  • A.2 Call Accuracy: A generated prediction passes the call stage only when it is non-empty, exports a valid AST-checked kernel entry, and executes successfully under the call harness.Quantization additionally requires a static quantization check at this stage.
  • A.3 Execution Accuracy: Execution accuracy requires recursive comparison against a reference under a shared random seed, with exact shape and dtype agreement before numerical comparison.Both generated and reference implementations must expose a test_results object.

B.1 Full Category-Level Results … D Analysis: Why LLMs Cannot Reliably Generate High-Performance Kernels

KernelBenchX’s category-level results show that correctness failures are structural rather than reducible to superficial code-complexity proxies. Quantization exposes a distinct semantic boundary, while the appendix analyzes failures through training data, prompt construction, and iterative feedback.

  • B.1 Full Category-Level Results: Tables 3 and 4 report semantic correctness and compile rates for every method across all benchmark categories.The results are organized at category level for both correctness and compilation.
  • B.2 Static Structure Proxies: All static-structure proxies correlate only modestly with correctness failure.The proxies are more predictive of compile failure than semantic failure.
  • B.2 Static Structure Proxies: Static structure proxies predict compile failure better than semantic failure, indicating that correctness is structural but not reducible to reference-code complexity.The analysis uses task-level static structure measures and Pearson correlations with pooled correctness failure.
  • C Quantization Details: 0% correctness on quantization, despite non-trivial compilation, reveals a distinct semantic boundary.The six quantization tasks require explicit scale computation and discretization rather than forbidden high-level quantization APIs.
  • C Quantization Details: Quantization failures indicate that models do not reliably preserve the numerical contract as part of the computation.All six quantization tasks use custom execution-stage precision thresholds.
  • D Analysis: Why LLMs Cannot Reliably Generate High-Performance Kernels: The appendix analyzes hardware-inefficient Triton generation through training data, prompt construction, and iterative feedback.These are presented as three structural factors behind current LLM-based methods’ failures.

D.1 Training Data Lacks Performance Grounding … D.4 Potential Improvement Directions

The discussion attributes weak GPU-kernel performance to missing hardware grounding in training, prompts, and iterative feedback, which emphasize syntax and correctness over transferable efficiency. It proposes profile-guided tuning and hardware-aware training as complementary remedies.

  • D.1 Training Data Lacks Performance Grounding: Code-corpus training teaches syntactic patterns but provides no performance annotations or hardware-behavioral representation.Models may generate kernels that look efficient without understanding why performance changes across hardware conditions.
  • D.1 Training Data Lacks Performance Grounding: Execution-based supervision improves constraint satisfaction but remains correctness-oriented rather than performance-oriented across hardware platforms.Methods such as AutoTriton reward execution and test success, allowing efficient-looking implementations without transferable hardware-performance understanding.
  • D.2 Prompt Construction Omits Hardware Context: None of the evaluated methods explicitly receive hardware information, limiting reasoning about shared memory, register spilling, and memory coalescing.Generated kernels therefore implicitly target average or prototypical hardware rather than a specified device.
  • D.3 Iterative Feedback Cannot Drive Performance Optimization: Compilation errors and correctness failures provide local feedback but cannot reveal whether kernels are memory-bound, compute-bound, or resource-constrained.Performance optimization requires information about hardware scheduling and execution-resource allocation that these signals do not recover.
  • D.3 Iterative Feedback Cannot Drive Performance Optimization: GEAK edit analysis shows that rescuing additional correct kernels lowers average speedup because newly rescued kernels underperform kernels correct from the start.The passage identifies this asymmetry through adjacent-diff analysis and cites Case 3 as an example.
  • D.3 Iterative Feedback Cannot Drive Performance Optimization: Non-local structural changes such as retiling, restructuring reductions, and reconsidering kernel boundaries are needed for substantive performance gains.These changes are unreachable through local neighborhood search from a correct but inefficient kernel.
  • D.4 Potential Improvement Directions: Profile-guided hyperparameter search and hardware-aware training could improve efficiency by tuning execution parameters and linking implementation choices to cross-platform outcomes.The proposed parameters include block size, warp count, and pipeline stages; training would pair hardware specifications with performance results.

E Artifacts: Transition Pairs for Repair and Optimization Analysis

The artifact set captures iteration-level transition events between adjacent GEAK rounds, supporting analysis of repair and optimization behavior. It includes correctness transitions, performance improvements, and regressions.

  • Artifact set: The artifact set centers on transition events between adjacent GEAK rounds.These records enable iteration-level analysis beyond aggregate results.
  • Correctness transitions: Correctness transitions record incorrect-to-correct changes with pass indicators, runtime and speedup when available, aligned errors, and unified diffs.Together, these fields form an explicit error →patch →pass chain for local repair modeling.
  • Performance events: Performance events capture rounds with execution-time improvements over the preceding round.These events were relatively infrequent in the experiment.
  • Regression events: Regression events capture rounds where a previously passing metric fails.Regression events were relatively infrequent in the experiment.
Loading 2605.04956v2…