PaperlayerView exact revision on arXiv

Source-linked AI summary

KernelFoundry: Hardware-aware evolutionary GPU kernel optimization

Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk, Benjamin Ummenhofer

arXiv:2603.12440v2cs.DCcs.LG

TL;DR

LLM-generated GPU kernels require hardware and parallel-programming expertise, while existing methods largely rely on standard prompting and feedback loops. KernelFoundry uses hardware-aware evolutionary search with prompt co-evolution and parameter optimization, and it reports strong CUDA and SYCL results, including an average SYCL speedup of 2.32 on the evaluated KernelBench setting.

  • Problem

    GPU kernel generation requires hardware and parallel-programming expertise, while most existing LLM approaches rely on standard prompting and feedback loops.

  • Method

    KernelFoundry combines kernel-specific MAP-Elites search, meta-prompt evolution, and templated parameter optimization for CUDA and SYCL kernels.

  • Results

    97% correctness and 2.32 average speedup were achieved for SYCL kernel generation on the evaluated KernelBench setting.

  • Takeaways & Limitations

    The framework supports cross-platform kernel optimization and user-defined workloads beyond standardized benchmarks.

Abstract

from arXiv · show

GPU kernel optimization challenges LLMs beyond standard coding tasks, as it requires an understanding of hardware architecture, parallel computing optimization strategies, and profiling outputs. However, most existing approaches leveraging LLMs for kernel generation apply standard prompting and feedback loops, considering hardware only through profiling feedback. We introduce KernelFoundry, an evolutionary framework that efficiently explores the space of GPU kernels through (1) MAP-Elites quality diversity search with kernel-specific behavioral dimensions to sustain exploration; (2) meta-prompt evolution that co-evolves prompts with kernels to uncover task-specific optimization strategies, and (3) a template-based parameter optimization approach to tune kernels to inputs and hardware. We evaluate this framework on Kernel-Bench, robust-kbench and custom tasks, generating SYCL kernels as a cross-platform GPU programming paradigm, and CUDA kernels for comparison to prior work. Our approach consistently outperforms the baseline methods and achieves an average speedup of 2.3 on KernelBench for SYCL. Moreover, KernelFoundry is implemented as a distributed framework with remote access to diverse hardware, allowing quick benchmarking and featuring a flexible user input layer to support kernel generation for a wide range of real use cases beyond benchmarking.

1. Introduction

KernelFoundry addresses the expertise and exploration challenges of LLM-generated GPU kernels with an evolutionary framework that combines hardware-aware diversity, prompt co-evolution, and parameter tuning. It is evaluated across standard and custom workloads, including cross-platform SYCL and CUDA comparisons.

  • GPU kernel optimization requires expertise in hardware architectures, memory hierarchies, and parallel programming.
  • Most LLM kernel-generation approaches use iterative generate→verify→measure loops to steer subsequent generations.
  • KernelFoundry maintains a diverse archive and uses MAP-Elites with kernel-specific behavioral dimensions to explore optimization strategies.
  • Meta-prompt evolution co-evolves prompts with kernels, while templated kernels tune hardware-dependent values such as tile and block sizes.
  • The evaluation spans KernelBench, robust-kbench, and custom suites covering operators, fusion patterns, and full model architectures.

2. Related Work

Prior work spans traditional scheduling systems, LLM-based kernel generation, finetuning, portability efforts, and evolutionary search. KernelFoundry extends these directions toward kernel-specific evolutionary optimization and SYCL-based portability.

  • Halide and TVM separate algorithm specification from optimization schedules, whereas CUDA and SYCL combine both in code; Triton provides an intermediate DSL.
  • KernelBench established a 250-task benchmark spanning single operators, fusion patterns, and complete architectures, while frontier models achieved correctness on roughly 20% of problems.
  • Finetuning methods use supervised learning, reinforcement learning, profiler metrics, or multi-turn refinement, but often cannot match the latest closed-source LLMs.
  • Existing LLM kernel work targets CUDA almost exclusively, while translation systems preserve rather than optimize performance.
  • Quality-diversity methods maintain diverse high-performing solutions in behavioral cells, motivating MAP-Elites for kernels with multiple valid implementation strategies.

3. Method

KernelFoundry combines flexible task specification, kernel-specific MAP-Elites search, correctness-first fitness, gradient-informed steering, parameter exploration, and meta-prompt evolution. These components preserve behavioral diversity while guiding kernels toward higher performance.

  • Task specification: The framework accepts PyTorch references, natural-language descriptions, existing kernels, and custom workloads with user-defined test frameworks.
  • MAP-Elites search: Quality-diversity search maintains diverse solutions in behavioral cells, helping the search escape local optima.
  • Kernel-specific descriptors: Kernel-specific descriptors classify memory access, algorithmic structure, and parallelism coordination into a 4^3 = 64-cell behavioral grid.
  • Fitness: The fitness function assigns zero to compilation failures, 0.1 to incorrect kernels, and 0.5 + 0.5 · s_norm to correct kernels.Here s_norm = min(1, speedup/target), with a default target of 2× over PyTorch.
  • Selection: Parent selection mixes uniform, fitness-proportionate, curiosity-driven, and island-based strategies to balance diversity, exploitation, and migration.
  • Gradient-informed evolution: Transition history yields fitness, improvement-rate, and exploration gradients that steer parent sampling and generate natural-language mutation hints.
  • Parameter optimization: Templated kernels expose hardware-dependent parameters such as work-group dimensions, tile sizes, and unroll factors for explicit exploration.
  • Meta-prompting: Meta-prompt evolution co-evolves prompt regions with kernels so successful strategies propagate and unsuccessful guidance is pruned.

4. Experimental setup

The experiments control hardware, programming paradigm, benchmarks, baselines, and correctness criteria to address hardware-dependent comparisons. They emphasize SYCL on Intel GPUs while also including CUDA comparisons and stricter correctness evaluation.

  • Comparisons vary with hardware, programming approach, and LLM ability, so the study controls these factors across benchmarks, hardware types, and languages.
  • Experiments use SYCL on Intel Arc 140V and Battlemage B580 GPUs, plus CUDA on an NVIDIA RTX A6000.
  • KernelBench contains 250 tasks, while filtering removes flawed or inefficient-baseline tasks to produce 111 tasks and a representative 40-task subset.
  • Baselines include PyTorch eager and torch.compile, but hardware dependence and reward-hacking-prone tasks limit comparability with aggregated prior results.
  • Correctness uses relative precision ν and requires ν < 0.01 for 99% of output values, supplemented by cosine similarity of flattened outputs.

5. Results

KernelFoundry consistently outperforms comparison methods across CUDA and SYCL kernel-generation evaluations, while also producing hardware-aware and practically competitive kernels. Its evaluations cover baseline comparisons, cross-GPU optimization, oneDNN comparisons, and a Llama 3 operation case study.

  • Baseline comparison: KernelFoundry’s CUDA kernels consistently outperform published baselines, achieving average speedups of 1.24 on KernelBench L1 and 2.1 on L2.The corresponding AI CUDA Engineer results are 1.01 and 1.61, respectively.
  • SYCL kernel generation: 97% correctness and 2.32 average speedup demonstrate strong SYCL kernel-generation performance despite SYCL being less familiar to LLMs than CUDA.OpenEvolve reaches comparable speedup only after 40 iterations and has a notable performance gap after 10 iterations.
  • Hardware awareness: 70% of kernels outperform counterparts optimized on another GPU, with average speedups of 1.537 for LNL-optimized kernels and 1.109 for BMG-optimized kernels.The crossover experiment benchmarks kernels optimized on Intel Arc B580 and Intel Arc 140V GPUs against each other.
  • Beyond PyTorch Eager: Generated SYCL kernels are competitive with highly optimized oneDNN implementations and provide a speedup in three cases.The comparison uses oneDNN’s C++ API and applies operator fusion where possible.
  • Llama 3 case study: KernelFoundry finds a correct rotary positional embedding kernel in two iterations, reaches 7.9× speedup within ten iterations, and reduces Llama 3.2 1B forward-pass time by 8%.On an Intel B580 GPU, total forward-pass time decreases from 0.413 s to 0.38 s.

6. Conclusion

KernelFoundry combines evolutionary search, meta-prompting, and parameter optimization to generate high-performing CUDA and SYCL kernels, with cross-platform applicability and reported superiority over state-of-the-art methods.

  • KernelFoundry combines kernel-specific quality-diversity search, meta-prompting, and parameter optimization for GPU kernel optimization.The framework generates high-performing CUDA and SYCL kernels.
  • The framework demonstrates cross-platform applicability and outperforms state-of-the-art methods.
  • KernelFoundry is intended to support automated, robust kernel optimization and adaptation to new hardware platforms.

Impact Statement

The work targets the expertise and tooling barriers in high-performance GPU kernel development through automated, cross-platform kernel generation and a Python-based implementation stack.

  • Impact Statement: High-performance GPU kernel development requires scarce expertise in hardware architectures, memory hierarchies, and parallel programming.
  • Impact Statement: KernelFoundry enables automated cross-platform GPU kernel generation across diverse hardware ecosystems.
  • Implementation: The framework uses OpenAI API, Anthropic API, and vLLM for LLM inference.
  • Implementation: Its compilation stack includes Intel oneAPI DPC++ 2025.2, NVIDIA CUDA Toolkit 12.8, and Triton 3.5.
  • Implementation: Profiling uses unitrace 2.3 for SYCL and NVIDIA Nsight Compute 2025.3.0.0 for CUDA.

B.2. Benchmarking kernel runtime

KernelFoundry benchmarks kernels with adaptive trial counts, isolates execution timing where possible, and supplements runtime measurements with correctness and profiling information.

  • Runtime measurement: Initial runtime trials determine warmup and main-trial counts using a minimal total measurement time.This adapts the number of trials to kernel speed rather than fixing it in advance.
  • Runtime measurement: Backward-operation measurements in robust-kbench include torch.autograd overhead, so KernelFoundry separately measures isolated kernel runtime for greater stability.
  • Profiling: Optional profiling reports execution time, memory bandwidth, compute utilization, and memory-bound versus compute-bound bottlenecks.
  • Configuration: Experiment-specific iteration counts and population sizes are configured through default hyperparameters.

C. Distributed System & Custom Task Input

KernelFoundry distributes language-model inference, compilation, execution, and database services across workers, while its custom task format supports configurable integration with existing projects.

  • Distributed infrastructure: The pipeline uses a lightweight main thread connected to four servers that can run locally or across remote machines.
  • Distributed infrastructure: The LLM server hosts generation models through cloud REST APIs or local vLLM instances.
  • Distributed infrastructure: Compilation workers execute DPC++ and nvcc toolchains without requiring a GPU.
  • Distributed infrastructure: Execution workers run correctness tests and benchmarks on GPU nodes with single-task-per-GPU isolation through a task queue.
  • Distributed infrastructure: A database server stores generated kernels, evaluation results, and evolutionary state for reproducibility and analysis.
  • Distributed infrastructure: Load balancing and queues allow worker scaling and increased parallelism.
  • Custom task input: Custom tasks use YAML configuration, Python build and test modules, and language-specific generated-code files with marked sections.

D. KernelBench task filtering

The KernelBench evaluation filters flawed tasks and selects representative tasks with diverse operations and verified baseline kernels. The associated prompts require functional, compilable GPU code, expose optimization guidance, and support templated parameter tuning.

  • Task filtering: 223 tasks would remain if filtering used criteria (1), (2), and (4), rather than also excluding low output standard deviation or baseline inefficiency.The authors consider criterion (3) too strict and note that baseline inefficiency does not necessarily invalidate comparisons.
  • Representative task selection: 40 representative tasks are selected across levels L1 and L2, with 20 tasks per level covering operations such as matmul, convolution, transposed convolution, and activations.Selection emphasizes diversity and excludes tasks classified as compromised.
  • Representative task selection: 53 of 229 published baseline kernels failed the stricter correctness tests used for selecting fair comparison tasks.The retained tasks require an available baseline kernel whose fastest published version passes the authors’ tests.
  • Prompt and evaluation setup: The main prompt requires a functional kernel matching the reference, efficient GPU execution, complete code, compilation, and full execution without cached results.It also incorporates reference code, tested kernels, runtime logs, hardware specifications, and optimization strategies.
  • Template-based optimization: KernelFoundry prompts models to convert tunable choices such as block sizes into template parameters and dispatch selected instantiations through a forward function.The templated implementation preserves functionality and exposes a Pybind11 forward interface for evaluation.

F.5. Evaluation of hardware-awareness

Cross-hardware evaluation indicates that kernels optimized for a target GPU generally outperform kernels optimized on another GPU when tested on the target hardware.

  • Hardware-aware optimization: Target-GPU optimization generally outperforms cross-optimized kernels on the target hardware, with hws > 1 indicating lower target-GPU runtime.The crossover experiment optimizes kernels separately on Intel B580 and Intel LNl, then evaluates each on both devices.

G. Results for open-source model (GPT-OSS 20B)

The open-source GPT-OSS 20B evaluation shows that limited model capability can prevent correctness, while successful runs still improve kernels and prompt evolution continues to refine guidance over iterations. Prompt changes concentrate on optimization strategies and common pitfalls, with recurring structures emerging over time.

  • Reproducibility results: 7 of 20 KernelBench level-2 cases failed to produce correct kernels after 40 iterations with a population of 4 per generation.Among successful cases, the generated kernels were optimized.
  • Convergence: KernelFoundry continues discovering higher-performing kernels beyond 80 iterations, particularly when meta-prompting is enabled.Per-iteration speedup can dip during exploratory search, whereas cumulative performance continues converging.
  • Prompt evolution: Prompt evolution produces both substantial changes and clustering, with similarities recurring between early and much later iterations.These recurring patterns suggest convergence toward stable prompt structures.
  • Qualitative prompt analysis: Optimization-strategy and common-pitfall sections change most often, while optimization philosophy and analysis guidance remain comparatively stable.Added guidance includes anti-pattern fixes and context-aware recommendations such as shared-memory convolution for small kernels with high tile reuse.
Loading 2603.12440v2…