PaperlayerView exact revision on arXiv

Source-linked AI summary

FastKernels: Benchmarking GPU Kernel Generation in Production

Gabriele Oliaro, Yichao Fu, May Jiang, Owen Lu, Junli Wang, Zhihao Jia, Hao Zhang, Samyam Rajbhandari

arXiv:2605.23215v1cs.LGcs.AIcs.CL

TL;DR

GPU kernel agents can perform well on isolated benchmarks without transferring those gains to production inference systems. FASTKERNELS addresses this gap by combining a production-grade inference framework with compositional, production-aligned evaluation, and reports sub-unity aggregate speedups for current agents under these conditions.

  • Problem

    Existing GPU kernel benchmarks use isolated kernels, synthetic inputs, simplified interfaces, and limited compilation-stack context, weakening evidence about production inference performance.

  • Method

    FASTKERNELS derives tasks from real architectures, matches production module interfaces, captures production tensors, and runs kernels within a compositional inference framework.

  • Results

    Current agents achieve sub-unity aggregate speedups over production baselines on FASTKERNELS, while the benchmark covers 96.2% (409/425) of HuggingFace architectures.

  • Takeaways & Limitations

    FASTKERNELS supplies a production-aligned evaluation surface intended to make benchmark outcomes more relevant to deployed inference systems.

  • Takeaways & Limitations

    The agent study uses only L1+L2, coverage is an indicative upper bound, and performance rankings may differ across hardware.

Abstract

from arXiv · show

LLM-based agents for GPU kernel generation are advancing rapidly, yet their progress is fundamentally constrained by the benchmarks they optimize against. Existing benchmarks are poorly aligned with production inference frameworks: they evaluate kernels on a single GPU with synthetic inputs, ignore the surrounding compilation stack, and reward replicating known optimizations rather than discovering new ones. The resulting reward signals are misleading: agents learn to generate kernels that score well in sandboxes but introduce interface incompatibilities, compilation-stack conflicts, and silent correctness degradation when integrated into real systems. We introduce FastKernels, a kernel benchmark built around a minimal set of 46 representative architectures spanning 8 categories, whose kernels collectively subsume those of 96.2% (409/425) of HuggingFace Transformers architectures. FastKernels doubles as a minimalistic, production-grade inference framework that runs at parity with hardened systems such as vLLM and SGLang on mainstream LLM serving and substantially exceeds upstream references on under-served architectures; each task's interface mirrors the corresponding module in the state-of-the-art library for its architecture family, enabling direct deployment of optimized kernels into production codebases. Evaluating state-of-the-art kernel agents on FastKernels, we find that even the strongest agent achieves only 0.94$\times$ aggregate speedup over production baselines, with weaker agents at $0.78\times$ and $0.53\times$ -- confirming that benchmark-production misalignment is a critical bottleneck for the field. We release FastKernels as a stepping stone toward kernel agents whose benchmark gains translate directly into production throughput improvements. Code is available at https://github.com/Snowflake-AI-Research/fastkernels

1 Introduction

FASTKERNELS addresses benchmark–production misalignment by combining a production-grade inference framework with production-faithful, compositional kernel evaluation across broad architecture coverage.

  • Existing kernel benchmarks often reward isolated speed while missing production interfaces, compilation effects, workloads, and model-scale correctness degradation.
  • FASTKERNELS unifies benchmarking and inference so generated kernels run inside real serving pipelines and can transfer to systems such as vLLM and SGLang.
  • Its compositional hierarchy connects primitives, fused operators, layers, and models, allowing agents to reuse lower-level optimizations in higher-level modules.
  • FASTKERNELS evaluates kernels against production baselines using captured tensors, compilation-stack effects, and multi-GPU communication patterns.
  • End-to-end validation checks downstream quality and combines correctness, coverage, and throughput–latency speedup through MACROEVAL.
  • 46 architectures across 8 categories provide broad coverage while tasks mirror production module interfaces and target real model components.

2 Related Work

Prior GPU kernel benchmarks and agents largely operate at the isolated, single-GPU operator level, leaving production-aligned evaluation gaps that FASTKERNELS targets.

  • GPU kernel benchmarks: Existing benchmarks span robustness, speed-of-light comparisons, inference-engine integration, and backend coverage but generally retain isolated operator-level evaluation.
  • GPU kernel benchmarks: FASTKERNELS adds multi-GPU communication kernels and a compositional hierarchy from primitives to full models.
  • GPU kernel benchmarks: It measures speedup against kernels shipped in state-of-the-art inference frameworks and matches production module interfaces for deployment.
  • LLM-based kernel agents: Kernel-agent systems combine LLM reasoning with profiler feedback, evolutionary search, reinforcement learning, or fine-tuning, typically using operator-level benchmarks.
  • LLM-based kernel agents: FASTKERNELS provides these agents with a production-aligned evaluation surface complementary to existing agent advances.

3 Benchmark Design

FASTKERNELS constructs a production-oriented benchmark top-down from real model architectures, organizing compositional tasks across four levels and covering broad architecture families. Its interfaces mirror production modules, while multi-GPU communication tasks and end-to-end deployment connect kernel optimization to real inference systems.

  • Top-Down, Model-Driven Construction: FASTKERNELS derives kernels from real model families by recursively decomposing their inference paths rather than assembling synthetic or automatically extracted tasks.Every task corresponds to an operation executed during real-model inference.
  • Task Hierarchy: FASTKERNELS organizes tasks into four levels: primitive operators, fused operators, full layers or blocks, and end-to-end model architectures.Levels 1–3 support isolated optimization, while Level 4 evaluates integrated model execution.
  • Architecture Coverage: 46 architectures span 8 categories and precision formats from 1.58-bit through FP32, with the set covering 96.2% (409/425) of HuggingFace Transformers architectures without a new compute primitive.The coverage audit found only 5 architectures requiring a genuinely new kernel and 2 requiring an external library.
  • Interface-Compatible Design: FASTKERNELS designs task constructors and forward methods to closely match corresponding modules in production libraries such as vLLM and SGLang.Optimized modules can follow either direct deployment in FASTKERNELS or transfer deployment into an existing production framework.
  • Multi-GPU Communication Kernels: FASTKERNELS includes tensor-parallel, expert-parallel, and overlap kernels because collectives, synchronization, and communication–computation overlap affect end-to-end latency.A kernel that is faster on one GPU can still reduce end-to-end throughput by disrupting the communication schedule.

4 Benchmarking Stack

FASTKERNELS exposes one evaluation stack through three tiers that range from kernel diagnostics to standardized end-to-end agent evaluation. Production-derived workloads, fixed configurations, subprocess isolation, and integrated profiling connect isolated measurements to deployment behavior.

  • Tier 1: Tier 1 runs candidate operators beside baseline modules and compares forward outputs and runtimes across production-derived shapes, dtypes, and initialization arguments.Its input registry is derived from Tier 3 workloads spanning multiple models, batch regimes, and tensor-parallel degrees.
  • Diagnostic Tiers: Tier 1 and Tier 2 are diagnostic tools for inspecting kernels, isolating regressions, and testing deployment-specific workloads.Tier 1 targets kernel-level analysis, while Tier 2 targets end-to-end execution traces.
  • Tier 3: Tier 3 compares baseline and candidate executions across fixed models, tensor-parallel configurations, and throughput and latency workloads.It is the comprehensive benchmark for LLM kernel agents and produces the leaderboard metrics.
  • Profiling: Profiling is integrated through NVIDIA Nsight Compute for Tier 1 and NVIDIA Nsight Systems for Tier 2, with data capture, parsing, and bottleneck extraction.Tier 1 uses production-derived shapes and real captured tensors for data-dependent operators.

5 MACROEVAL: Cross-Architecture Metrics

MACROEVAL evaluates candidate kernels inside end-to-end model executions using architecture-specific correctness signals and runtime measurements. It calibrates correctness to a common scale and macro-averages across families so heterogeneous architectures contribute comparably.

  • Evaluation Design: MACROEVAL substitutes candidate kernels into end-to-end model executions and compares them with reference runs using production baseline kernels.The framework must accommodate outputs including tokens, embeddings, labels, rankings, trajectories, audio, and video.
  • Cross-Architecture Metrics: MACROEVAL combines calibrated correctness, coverage, and end-to-end throughput–latency speedup across model families.The metric is designed to prevent any single architecture from dominating the leaderboard.
  • Calibrated Correctness: Architecture families use distinct discrepancy measures, including token divergence, embedding distance, IoU, WER/CER, ranking disagreement, and trajectory error.The family-specific discrepancy D_f is computed from reference and candidate outputs for each request.
  • Calibrated Correctness: Raw discrepancies are mapped to calibrated correctness scores in [0, 1] using thresholds that distinguish reference-level nondeterminism from incorrect or unusable outputs.Thresholds are fixed before evaluation and released with the benchmark; g_i marks indistinguishability, while f_i marks the incorrectness boundary.

Appendix C.

FASTKERNELS evaluates kernel agents with separate macro-averaged measures for correctness, validity-aware coverage, and production-relative speedup, then combines them in a default ranking. The design preserves deployment-relevant tradeoffs by withholding speedup credit from invalid items and exposing alternative workload-specific rankings.

  • Correctness: Macro-averaged correctness gives each architecture family equal influence, preventing overrepresented families such as LLMs from dominating the correctness axis.
  • Validity and coverage: Items are invalid when they crash, hang, trigger shape or type errors, access memory illegally, produce NaNs, or fall below a correctness threshold.Validity uses a pre-specified item or family threshold.
  • Validity and coverage: Macro-coverage reports the fraction of valid items across architecture families and is preferred when architecture balance matters.
  • Throughput–latency speedup: Valid items receive throughput and latency speedups against the production reference, while invalid items receive no speedup credit.
  • Throughput–latency speedup: The balanced geometric blend uses λ = 0.5, so equivalent throughput gains and latency degradations receive no net performance credit.Users may choose a different pre-specified λ for throughput- or latency-sensitive settings.
  • Leaderboard: Families without valid items are excluded from speedup aggregation but penalize the agent through macro-coverage, preventing correctness-for-speed tradeoffs.Where sample sizes permit, 95% bootstrap confidence intervals quantify variability under noisy serving conditions.
  • Leaderboard: The default score is an equally weighted product of macro-geometric speedup, macro-calibrated correctness, and macro-coverage.The leaderboard also exposes the three components and reports throughput-only and latency-only rankings.

6 Evaluation

FASTKERNELS evaluates kernels against production-shaped references on real workloads, exposing gaps that isolated benchmarks miss. Existing agents often lose speedup and correctness when moving from primitives to production-composite modules.

  • Reference performance and correctness: 1.24× average throughput and 1.04× median show FASTKERNELS is broadly competitive across 46 representative benchmarks.Mainstream LLM serving remains near parity with hardened frameworks, while larger gains concentrate on under-served architectures.
  • Agent evaluation: Codex reaches 1.16× geomean on 22 torch-eager-wrapped families but only 0.93× on 26 vendor- or hand-written kernels.Anchoring comparisons to deployable production kernels removes most apparent operator-level gains.
  • Agent evaluation: Production references expose losses on hot paths, including moe_align at 0.28×, linear at 0.56×, and flashinfer decode/prefill at 0.71×.Gains above 1× cluster on operators without specialized production implementations.
  • Agent evaluation: Allreduce exposes a correctness failure: a Triton copy passes a single-process checker but produces the wrong sum in every 4-rank NCCL+Gloo scenario.The staged harness reduces all-reduce to identity, masking distributed communication behavior.
  • Agent evaluation: L2 composite tasks cause universal regressions, with KernelAgent dropping from 26/40 correct at 0.79× to 2/36 at 0.629× and Codex from 1.03× to 0.84×.Failures commonly violate surrounding production contracts despite respecting per-tensor signatures.

7 Conclusion

FASTKERNELS grounds kernel-generation evaluation in production inference through compositional tasks, matched interfaces, captured tensors, and deployed-baseline references. Its evaluation shows that current agents remain below production parity, motivating benchmarks whose gains transfer to throughput.

  • Conclusion: FASTKERNELS is a benchmark-as-framework that evaluates generated kernels inside production-shaped inference pipelines rather than isolated sandboxes.Its design combines top-down tasks, production-matched interfaces, captured tensors, and deployed-baseline references.
  • Conclusion: Current agents fall below parity with production kernels, with gains concentrated on operators lacking specialized production implementations.The conclusion identifies benchmark–production misalignment as a concrete cost of current evaluation practice.
  • Limitations: The agent study covers L1 and L2 only; L3/L4 are shipped but excluded from headline numbers.The 96.2% coverage is an indicative upper bound, hardware results come from H100 SXM5, and MACROEVAL choices encode value judgments.

A Benchmark Composition

FASTKERNELS organizes its benchmark around architecture-specific task compositions spanning four levels. The tables document each architecture’s task counts, reference checkpoint or repository, and default inference dtype.

  • Benchmark composition: Table 2 reports each architecture’s reference checkpoint, default inference dtype, and counts of L1–L4 tasks.The levels correspond to primitive, fused/composite, layer/block, and end-to-end tasks.
  • Task levels: L1 denotes primitive tasks, L2 fused/composite tasks, L3 layer/block tasks, and L4 end-to-end tasks.L4 is always one task by construction.
  • Task levels: A zero task count means the current implementation exposes no benchmark task at that level.Quantized configurations reuse the underlying architecture’s high-level graph while activating distinct kernels and execution paths.

B Interface Compatibility Examples

FASTKERNELS mirrors production module interfaces so optimized kernels can substitute directly into inference code. Examples cover attention and MoE modules with matching constructors and forward arguments.

  • Interface compatibility: Side-by-side examples compare FASTKERNELS interfaces with corresponding vLLM modules.The comparison demonstrates the benchmark’s interface-compatible design.
  • Attention: The attention example matches __init__(config, cache_config, quant_config) and forward arguments positions, hidden_states, kv_cache, and attn_metadata.The corresponding classes are Attention and LlamaAttention.
  • Interface compatibility: Matching constructor signatures and forward arguments enable direct module substitution across attention, MLP, normalization, and MoE routing modules.The same compatibility pattern extends beyond the illustrated attention example.
  • MoE: The MoE example pairs MoELayer and DeepseekV3MoE with matching config initialization and hidden_states, router_logits inputs.These signatures illustrate compatibility for routing modules.

C MACROEVAL Calibration and Sensitivity

MACROEVAL calibrates correctness thresholds from reference nondeterminism and quality cliffs, then tests sensitivity to tolerance and harness-gap accounting. Bootstrap intervals quantify uncertainty in agent speedups and rankings.

  • Calibration: Correctness bands use reference-vs-reference nondeterminism, while unacceptable thresholds follow family-specific quality cliffs from deliberately wrong baselines.The calibrated band is gi = atol + rtol · |yref|; thresholds and calibration values are frozen in the released manifest.
  • Sensitivity: Codex rankings are evaluated across tolerance scales from 0.25× to 5× of the default per-scenario error-ratio threshold.The 1× setting reproduces the headline, and loosening above 1× does not change the result because no scenario crosses the default error ratio.
  • Sensitivity: Harness-gap accounting separates coverage penalties from speedup aggregation, preventing blocked or incorrect targets from receiving performance credit.Blocked targets reduce coverage, while alternative policies bracket the headline geomean speedups.
  • Uncertainty: 95% bootstrap intervals confirm the combined Codex–KernelAgent ordering and Codex’s L1-to-L2 decline, while small samples limit separation of Dr. Kernel from KernelAgent.Dr. Kernel’s interval is wide because only eight correct kernels were sampled, including a heavy-tailed outlier.

D Case Study: Captured Inputs Matter for Data-Dependent Ops

Captured inputs materially alter data-dependent MoE execution: random tensors produce near-uniform routing, while random token IDs select different hot experts than real requests. FASTKERNELS therefore replays production-captured tensors for data-dependent kernels.

  • Routing patterns: Real WildChat requests, random token IDs, and random gate tensors produce different expert-routing patterns in Qwen3-VL’s first MoE layer.The setup uses 128 experts with top-8 routing.
  • Routing patterns: Only 4 of the top-16 hot experts overlap between random token IDs and real requests.Random token IDs have Gini 0.480 versus 0.339 for real requests; random tensors have Gini 0.232.
  • Benchmark design: Capture-and-replay evaluates data-dependent kernels on tensors generated by production requests rather than synthetic control flow.This design targets the execution patterns that real serving workloads create.

E Experimental Details

Experiments run on fixed H100 hardware with controlled serving workloads, perplexity evaluation, and published default configurations for the external kernel-generation agents.

  • Hardware: All experiments use NVIDIA H100 SXM5 80GB GPUs, with 8-GPU NVLink nodes for multi-GPU runs and clocks locked at 1,593 MHz.Single-GPU experiments use one GPU.
  • Serving benchmark: Serving throughput uses ShareGPT prompts with batch size 32, 512 input tokens, and 128 output tokens.Each measurement averages three runs after ten warmup iterations.
  • Perplexity evaluation: Perplexity is measured on a held-out 10K-token WikiText-103 subset using sliding windows with stride equal to maximum context length.
  • Agent configuration: Dr. Kernel, KernelAgent, and Codex run in their default published configurations for the external-agent evaluation.The internal Codex-based reference uses NCU feedback, ten optimization iterations, and temperature sampling.

F Reference Performance and Correctness Details

The reference-performance tables summarize representative end-to-end coverage across architectures using production or upstream baselines, public workloads, throughput ratios, and alignment values. Their arithmetic means follow upstream README conventions and are not directly comparable to MACROEVAL geometric means.

  • Aggregation: The reported arithmetic means are not directly comparable to MACROEVAL’s geometric-mean aggregations.Arithmetic means are used to match upstream README conventions, including the cross-architecture summary in Section 6.1.
  • Coverage tables: Tables 7 and 8 report representative end-to-end benchmark coverage for each FASTKERNELS architecture.Table 7 contains Part I and Table 8 contains Part II.
  • Table fields: Each row identifies a SOTA reference, public dataset or workload, arithmetic-mean throughput ratio, and alignment value when available.A single reported throughput scenario is shown directly, while unavailable scalar summaries are marked accordingly.

G HuggingFace Transformers Coverage

FASTKERNELS maps HuggingFace Transformers classes to composable kernel files and measures coverage across audited architectures. The audit finds broad coverage through existing primitives, while several architectures remain partial or unsupported because they require specialized operations.

  • Coverage: 96.2% (409/425) of audited HuggingFace architectures are covered without a new compute primitive.Only 5 architectures require a genuinely new kernel, while 2 require an external library; the headline is an indicative upper bound because covered verdicts were sampled rather than exhaustively re-audited.
  • Mapping: Table 9 maps each HuggingFace class to the FASTKERNELS pipeline and the kb-nano files it invokes.Wiring-only classes and duplicate signatures are collapsed, with wiring classes restored when they introduce otherwise uncovered kernels.
  • Residual cases: Autoformer is partial because its attention uses FFT-based autocorrelation and top-k delay aggregation, which has no kb-nano kernel.Its remaining modules use mapped embedding and pooling, normalization, linear, convolution, activation, and dropout kernels.
  • Composable coverage: Most covered architectures compose recurring primitives and fused operators across language, vision, audio, and multimodal model families.Examples include attention, linear, normalization, activation, convolution, embedding, and pooling kernels reused across model classes.
  • Residual cases: Reformer is unsupported because LSHSelfAttention is a bespoke LSH attention operation without a kb-nano kernel.Other Reformer components still map to tensor operations, embeddings, linear layers, dropout, normalization, and dense attention.
Loading 2605.23215v1…