Source-linked AI summary
Towards Automated Kernel Generation in the Era of LLMs
Yang Yu, Peiyu Zang, Chi Hsu Tsai, Haiming Wu, Yixin Shen, Jialing Zhang, Haoyu Wang, Zhiyou Xiao, Jingze Shi, Yuyu Luo, Wentao Zhang, Chunlei Men, Guang Liu, Yonghua Lin
TL;DR
Modern AI performance depends heavily on high-quality kernels, but producing them requires specialized expertise and remains difficult to scale. This survey unifies LLM-based generation and agentic optimization methods, organizes their datasets and benchmarks, and identifies challenges for reliable evaluation, data, training, and infrastructure. It positions these resources and directions as a reference for advancing automated kernel optimization.
Problem
Kernel engineering is expertise-intensive, hardware- and workload-specific, and the rapidly growing LLM-driven kernel-generation field lacks a systematic synthesis.
Method
The survey structures LLM-based approaches and agentic optimization workflows while consolidating kernel datasets, benchmarks, and a RAG-oriented literature resource.
Results
The survey synthesizes recent advances in LLM and agentic kernel generation and organizes the datasets and benchmarks supporting learning and evaluation.
Takeaways & Limitations
Future progress depends on reliable evaluation protocols, data infrastructure, and harness engineering for agentic kernel generation.
Takeaways & Limitations
Current evaluation is vulnerable to reward hacking and limited workload, platform, and execution coverage, while long-horizon agent training remains underdeveloped.
Abstract
from arXiv · showhide
The performance of modern AI systems is fundamentally constrained by the quality of their underlying GPU kernels, which translate high-level algorithmic semantics into low-level hardware operations. Achieving near-optimal kernels requires expert-level understanding of hardware architectures and programming models, making kernel engineering a critical but notoriously time-consuming and non-scalable process. Recent advances in large language models and LLM-based agents have opened new possibilities for automating kernel generation and optimization. LLMs are well-suited to compress expert-level kernel knowledge that is difficult to formalize, while agentic systems further enable scalable optimization by casting kernel development as an iterative, feedback-driven loop. Rapid progress has been made in this area. However, the field remains fragmented and lacks a systematic perspective for LLM-driven kernel generation. This survey addresses this gap by providing a structured overview of existing approaches, spanning LLM-based approaches and agentic optimization workflows, and systematically organizing the datasets and benchmarks that underpin learning and evaluation in this domain. Moreover, key open challenges and future research directions are further outlined, aiming to establish a comprehensive reference for the next generation of automated kernel optimization. To keep track of this field, we maintain an open-source GitHub repository at https://github.com/flagos-ai/awesome-LLM-driven-kernel-generation.
1 Introduction
Kernel quality strongly influences modern AI-system efficiency, yet kernel engineering remains expertise-intensive and difficult to scale. This survey organizes a fragmented field of LLM-driven generation and agentic optimization while consolidating resources and future directions.
- Kernel implementations for operations such as matrix multiplication and attention often determine AI-system throughput, efficiency, and cost.
- Near-peak kernel performance requires algorithmic and hardware expertise, while architecture- and workload-specific implementations limit reuse across platforms.
- LLMs encode hardware-aware programming expertise, and agents use iterative execution feedback to explore kernel optimizations across workloads and hardware platforms.
- The field has progressed rapidly but remains fragmented and lacks a systematic synthesis.
- The survey provides a unified overview, structured kernel datasets, a RAG-oriented literature collection, and open challenges and research directions.
2 Background
LLMs model token sequences through Transformer-based next-token prediction, while agents extend them with planning, memory, tools, and environmental feedback. Kernel generation combines expert- and compiler-driven optimization paradigms but demands specialized performance and hardware awareness.
- LLMs and LLM-based Autonomous Agents: Transformers serve as probabilistic predictors trained with a next-token objective over token sequences.
- LLMs and LLM-based Autonomous Agents: The next-token objective enables LLMs to internalize world knowledge and reasoning patterns implicitly during pretraining.
- LLMs and LLM-based Autonomous Agents: Autonomous agents add planning, memory, and tool use, interacting with environments through trial and error and feedback.
- Kernel Programming and Code Generation: Kernel development traditionally uses expert-written hardware abstractions or compiler-driven scheduling and autotuning, each with distinct constraints.
- Kernel Programming and Code Generation: Unlike general code synthesis, kernel generation must satisfy functional correctness, stringent performance requirements, and hardware-specific execution characteristics.
3 LLM for Kernels Generation
LLM-based kernel generation has developed two major post-training families: supervised fine-tuning on aligned code pairs and reinforcement learning with iterative feedback. Recent work applies these methods across CUDA, Triton, and Ascend kernels.
- Recent kernel-generation research commonly specializes LLMs through supervised fine-tuning or reinforcement learning.
- Supervised Fine-Tuning: Supervised fine-tuning uses paired datasets linking high-level computational intent with low-level kernel implementations.
- Reinforcement Learning: Reinforcement-learning methods formulate kernel generation as iterative optimization and use execution, contrastive, or model-based feedback for reward signals.
- Reinforcement Learning: Triton-focused methods combine structural assessments with runtime rewards, distributed environments, multi-turn training, and evolution-oriented post-training.
- Reinforcement Learning: Kernel-generation alignment has expanded beyond CUDA and Triton to Ascend NPUs and AscendC kernels.
4 LLM Agent for Kernels Generation
LLM agents turn kernel optimization into a closed feedback loop involving planning, tool use, and evaluation. The surveyed systems organize this automation around learning, memory, hardware profiling, evolution, and coordinated agent roles.
- Agentic Optimization: Agent-based kernel optimization adds planning, tool use, and intermediate evaluation to support closed-loop, long-horizon exploration.
- Iterative Refinement: Iterative-refinement agents generate and revise kernels using execution feedback, reflection, modular workflows, and coding-agent harnesses.
- Population-Based Evolution: Population-based methods use mutation, crossover, diversity preservation, adaptive evolution, and multi-population dynamics to escape local optima.
- External Memory: External-memory approaches provide hardware-specific knowledge and prior interaction experience to reduce reliance on unsupported or forgotten domain details.
- Hardware Profiling Integration: Hardware-aware agents inject architecture and instruction-set information while using compilation logs, runtime metrics, and profiling to guide optimization.
- Multi-Agent Orchestration: Agent systems increasingly decompose kernel development into coordinated planning, coding, debugging, judging, and platform-specific roles.
5 Datasets
The survey organizes kernel-generation resources into training corpora and knowledge bases, covering executable kernels alongside hardware and optimization knowledge.
- Resource Categories: Resources are divided into Training Corpora for model training and Knowledge Bases for domain-specific retrieval-augmented generation.Training corpora include structured datasets and kernel code repositories, while knowledge bases provide specialized knowledge.
- Resource Overview: Table 1 provides a structured overview of these training corpora and kernel knowledge bases, with dates indicating each resource’s initial release.The libraries themselves continue to undergo active development.
- Training Corpora: Structured datasets pair high-level intents with optimized implementations, whereas raw repositories collect expert-written kernels from libraries, frameworks, and domain-specific languages.
- Knowledge Bases: Knowledge bases supply hardware and optimization information through technical documentation, programming guides, tutorials, and community resources.This knowledge can be incorporated during pretraining or accessed dynamically through RAG.
6 Benchmark
Kernel-generation benchmarks evaluate correctness, performance, and composite quality, increasingly broadening their metrics, hardware coverage, and workload realism.
- Evaluation Protocol: Benchmarks typically execute generated kernels and compare them with reference CUDA or PyTorch implementations, repeating evaluations across samples and random seeds.This accounts for the stochastic nature of kernel generation.
- Metrics: Correctness measures output consistency with a reference within numerical tolerances that vary across kernel types and precision formats.Examples of evaluated formats include FP16, BF16, and FP8.
- Metrics: Performance is primarily measured as runtime speedup relative to a reference implementation, often using wallclock time averaged over repeated trials.Some benchmarks additionally compare results with Speed-of-Light estimates of theoretical hardware efficiency.
- Metrics: Composite metrics assess multiple quality dimensions, including Similarity and fastp, which measures the proportion of correct kernels exceeding a specified speedup p.
- Benchmark Trends: Benchmark evolution expands from correctness and speedup toward efficiency, robustness, and overall quality, while adding AMD GPUs, Huawei NPUs, Google TPUs, and production-grade workloads.Examples include TritonBench, MultiKernelBench, FlashInfer-Bench, and SOL-ExecBench.
7 Challenges and Opportunities
The survey identifies reliability, data, agent, infrastructure, and collaboration constraints that must be addressed for scalable LLM-driven kernel generation.
- Evaluation Reliability and Generalization: Existing evaluations remain vulnerable to reward hacking and limited generalization across workloads, hardware platforms, and execution settings.The impact of kernel-level improvements on end-to-end AI systems is also insufficiently understood.
- Data Scarcity and Synthetic Scaling: High-performance kernels are scarce in current corpora, which mostly contain final implementations rather than optimization trajectories and hardware-aware expertise.The survey points to larger datasets, synthetic data, and execution-driven optimization traces as directions.
- Agentic Training and Harness Engineering: Kernel optimization requires long-horizon cycles of generation, execution, profiling, and refinement, but current models and handcrafted workflows struggle with exploration, context, and credit assignment.Future progress may combine long-horizon agentic training with autonomous harness engineering.
- Evaluation: Efficiency is defined as the ratio of measured operator throughput to the theoretical maximum performance.
- Evaluation Scope: Benchmark datasets track correctness, speedup, efficiency, fastp, and similarity across NVIDIA GPUs, Huawei NPUs, Google TPUs, and AMD GPUs.
- Agentic Training and Harness Engineering: Foundation agents may be specialized through harness engineering using domain-specific environments, tools, and feedback rather than building task-specific agents from scratch.
- Scalable Infrastructure for Synthesis and Training: Scalable infrastructure must provide isolated distributed execution, reduce compilation and verification latency mismatches, and support fault-tolerant multi-device services.These capabilities are presented as necessary for systematic, data-driven kernel synthesis and agentic training.
- Human-AI Collaboration for Kernel Generation: Human-AI collaboration can combine human high-level objectives and constraints with agent-discovered knowledge transferred back to developers.The survey describes this as a potentially bidirectional co-evolutionary cycle.
8 Conclusion
The survey synthesizes LLM- and agent-based kernel generation methods with kernel-centric datasets and benchmarks, emphasizing reliable evaluation, data infrastructure, and harness engineering as future priorities.
- Conclusion: Future progress depends on reliable evaluation protocols, data infrastructure, and harness engineering for agentic kernel generation.The survey associates these advances with alleviating manual kernel-engineering burden and potential productivity gains for scaling AI infrastructure.