Source-linked AI summary
Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization
He Du, Qiming Ge, Jiakai Hu, Aijun Yang, Zheng Cai, Zixian Huang, Sheng Yuan, Qinxiu Cheng, Xinchen Xie, Yicheng Chen, Yining Li, Jiaxing Xie, Huanan Dong, Yaguang Wu, Xiangjun Huang, Jian Yang, Hui Wang, Bowen Zhou, Bowen Li, Qipeng Guo, Kai Chen
TL;DR
High-performance GPU kernel generation requires reliable iterative search rather than one-shot prediction, while training such optimization remains difficult. Kernel-Smith combines a stable evaluation-driven evolutionary agent with step-centric post-training, achieving strong KernelBench results, heterogeneous-backend validation, and production contributions. Its reinforcement-learning design also shows that selecting appropriate evolution steps matters for effective learning.
Problem
High-performance kernel generation remains difficult because effective optimization requires iterative search, reliable execution feedback, and training signals that support multi-step improvement.
Method
Kernel-Smith evolves populations of executable candidates using archived solutions and structured evaluation, while post-training retains correctness-preserving, high-gain evolution steps for local improvement.
Results
Kernel-Smith-235B-RL achieves state-of-the-art overall KernelBench performance and the best average speedup ratio, while the framework also transfers to MetaX and production systems including SGLang and LMDeploy.
Takeaways & Limitations
Stable execution feedback and step-centric training support kernel optimization beyond one-shot generation and across heterogeneous platforms and deployment settings.
Takeaways & Limitations
Using all evolution steps can enable information leakage, while using only the initial step yields suboptimal reinforcement-learning performance.
Abstract
from arXiv · showhide
We present Kernel-Smith, a framework for high-performance GPU kernel and operator generation that combines a stable evaluation-driven evolutionary agent with an evolution-oriented post-training recipe. On the agent side, Kernel-Smith maintains a population of executable candidates and iteratively improves them using an archive of top-performing and diverse programs together with structured execution feedback on compilation, correctness, and speedup. To make this search reliable, we build backend-specific evaluation services for Triton on NVIDIA GPUs and Maca on MetaX GPUs. On the training side, we convert long-horizon evolution trajectories into step-centric supervision and reinforcement learning signals by retaining correctness-preserving, high-gain revisions, so that the model is optimized as a strong local improver inside the evolutionary loop rather than as a one-shot generator. Under a unified evolutionary protocol, Kernel-Smith-235B-RL achieves state-of-the-art overall performance on KernelBench with Nvidia Triton backend, attaining the best average speedup ratio and outperforming frontier proprietary models including Gemini-3.0-pro and Claude-4.6-opus. We further validate the framework on the MetaX MACA backend, where our Kernel-Smith-MACA-30B surpasses large-scale counterparts such as DeepSeek-V3.2-think and Qwen3-235B-2507-think, highlighting potential for seamless adaptation across heterogeneous platforms. Beyond benchmark results, the same workflow produces upstream contributions to production systems including SGLang and LMDeploy, demonstrating that LLM-driven kernel optimization can transfer from controlled evaluation to practical deployment.
1. Introduction
Kernel-Smith addresses the difficulty of producing high-performance kernels through a unified evolutionary agent and evolution-oriented training recipe. Stable evaluation and step-centric learning support iterative improvement, yielding strong benchmark and deployment results.
- Motivation: High-performance kernel generation remains difficult because effective implementations require searching across fusion, tiling, and rewrite choices while preserving correctness.Existing refinement loops may anchor proposals to early decisions and limit exploration diversity.
- Framework: Kernel-Smith combines a reliable evaluation-driven evolutionary agent with training tailored to evolutionary search.The framework identifies key improvement steps from evolution trajectories.
- Agent: Stable evaluation uses fixed computation graphs, repeated measurements, and outlier removal to suppress timing noise and preserve reliable search dynamics.The agent maintains executable candidates whose performance gains accumulate over multiple search rounds.
- Training: Evolution-oriented post-training retains correctness-preserving, high-gain revisions so the model acts as a local improver rather than a one-shot generator.The approach converts long-horizon trajectories into step-centric training signals.
- Results: Kernel-Smith-235B-RL achieves state-of-the-art overall KernelBench performance, the best average speedup ratio, and accepted contributions to SGLang and LMDeploy.Its best-score curve remains above competing models throughout evolution, while the workflow transfers to production systems.
2. Related Work
Related work frames kernel generation as both a benchmarked performance-and-correctness problem and a structured search problem. Kernel-Smith builds on evolutionary and reinforcement-learning approaches to improve iterative optimization.
- Benchmarks: KernelBench evaluates replacing PyTorch implementations with faster GPU kernels using metrics that jointly reflect correctness and speedup.Pass rate alone does not capture whether generated kernels are practically useful.
- Benchmarks: Figure 1 compares best program score trajectories across evolutionary generations, with scores proportional to speedup and penalizing compilation or correctness failures.Kernel-Smith-235B-RL exhibits superior evolutionary efficiency and peak performance in the captioned comparison.
- Training: RLVR-based systems address scarce human-optimized CUDA and Triton data, but iterative optimization introduces context explosion and sparse reward attribution.Prior workflows use automated distillation, specialized agents, profiler feedback, or multi-turn reinforcement learning.
- Search: Recent systems treat the non-convex kernel landscape as structured search, using memory, exploration–exploitation strategies, contrastive learning, or targeted reinforcement learning.These approaches move beyond pure one-shot prediction.
3. Kernel-Smith
Kernel-Smith generates kernels by evolving executable candidates under structured, backend-specific evaluation. Its backend-decoupled design supports consistent optimization across NVIDIA and MetaX accelerators.
- Task: Kernel-Smith requires generated candidates to compile, match PyTorch numerical outputs, and improve execution speed over the eager baseline.The task combines functional preservation with hardware-specific efficiency.
- Agent: The evolve-agent maintains a population of candidate programs instead of refining one dialogue trajectory, broadening exploration of the kernel search space.An automated backend returns structured feedback on compilation, correctness, and performance.
- Agent: The evolutionary archive samples top-performing and diverse candidates, while structured feedback includes runtime, speedup, hardware metadata, and error logs.These signals expose both strong solutions and informative failure cases to later iterations.
- Backends: The evaluation service generates Triton kernels for NVIDIA GPUs and Maca kernels for MetaX GPUs under compilation, correctness, and speedup criteria.The distributed API provides parallel evaluation interfaces.
- Backends: Warm-up executions and noise-reduction procedures constrain execution-time fluctuations to within 1%.Timing stability is important because GPU wall-clock measurements can fluctuate even with fixed hardware and drivers.
- Backends: Runtime hacking detection prevents models from bypassing kernel generation by calling native PyTorch operators.The system also identifies trivial optimizations that pass tests but provide little practical engineering value.
- Heterogeneous Platforms: The backend-decoupled design separates task specification, orchestration, and metrics from device-specific compilation and runtime interfaces.The same protocol is instantiated for NVIDIA Triton and MetaX MACA and can extend to additional platforms.
4. Training Recipe
Kernel-Smith’s training recipe converts diverse PyTorch implementations and multi-step evolutionary trajectories into filtered, step-centric supervision and reinforcement-learning signals. Selecting high-value revisions improves iterative optimization while avoiding leakage and mismatched initial-step data.
- Training Data: The recipe converts multi-step evolution trajectories into individual improvement-step signals and retains high-gain, correctness-preserving revisions.This trains the model as a local improver inside the evolutionary loop.
- Training Data: Curated PyTorch data is sourced from diverse repositories, normalized through dependency resolution and import inference, then deduplicated using embedding- and graph-based methods.The process broadens coverage beyond canonical operators and standardized repository structures.
- Data Synthesis: Kernel-Smith synthesizes cold-start and cluster-seeded expert trajectories, filtering samples for functional validity and performance improvement.The two synthesis components use teacher rollouts and clustered data to raise training-data quality.
- Data Filtering: Correctness-oriented augmentation keeps functionally correct PyTorch-to-Triton translations, whereas performance-oriented augmentation retains Triton-to-Triton samples with speedup ratio > 1.0.The filters reflect the distinct objectives of translation and iterative optimization.
- Reinforcement Learning: Including all evolution steps can cause information leakage, allowing the model to memorize superior later-step kernels rather than learn generalized optimization.The resulting reward curve may look favorable while actual learning efficacy remains marginal.
- Reinforcement Learning: Selecting only the initial evolution step is suboptimal because it emphasizes simple PyTorch-to-Triton migration rather than substantial throughput acceleration.Its distribution differs from later optimization stages, making it unsuitable for effective reinforcement learning.
- Reinforcement Learning: Selecting the best evolution steps improves performance by constraining the learning space while retaining sufficient optimization challenge.Consistent multi-round inference gains indicate that these steps capture an atomic iterative capability.
- Expert Data: Cluster-seeded expert data uses 40 Gemini-3.0-pro refinement iterations and selects best steps to form training examples with high-performance kernel exemplars.The model modifies a designated parent kernel to produce a more efficient implementation.
5. Experiments
The experiments compare diverse models under a controlled Kernel-Smith evolutionary protocol on NVIDIA and MetaX kernel-generation tasks. Kernel-Smith achieves strong NVIDIA results and outperforms large-scale baselines on MetaX.
- Experimental setup: 40 evolutionary rounds are applied to every model under the same Kernel-Smith agent framework, with fixed decoding settings for fair comparison.The protocol uses temperature 0.6 and top-p 0.95.
- Metrics: The evaluation uses correctness, fast proportion, and average speedup-related metrics to assess generated operators.Correctness includes hack detection and requires precision differences to remain within an acceptable threshold.
- NVIDIA results: 96.33 average corr gives Kernel-Smith higher accuracy than Gemini-3.0-pro at 94.33 and DeepSeek-v3.2-Speciale at 94.67.Claude-4.6-opus remains highest on overall correctness with corr of 99.33.
- NVIDIA results: 3.70 average amsr is the highest overall score, while Level 2 reaches 7.77 versus Claude-4.6-opus at 5.83.On Level 3 tasks, Kernel-Smith sustains a correctness rate of 94.
- MetaX results: The MetaX benchmark contains 45 correctness-verified CUDA-to-MACA operator tasks across activation, normalization, reduction and aggregation, and loss-function groups.The benchmark includes 15 activation, 8 normalization, 17 reduction and aggregation, and 5 loss-function operators.
6. Real-world Applications
Kernel-Smith transfers its search-and-integration workflow from controlled operator benchmarks to production inference engines and recent research modules. The applications combine backend-specific kernel search with correctness validation, measurable speedups, and upstream integration.
- Workflow: Kernel-Smith applies the same workflow across SGLang, LMDeploy, and Engram by extracting target modules, testing candidates, and integrating successful kernels upstream.The process uses compilation, correctness, and speedup feedback before repository integration and supplementary testing when needed.
- SGLang: 4.78× speedup is reported for the fused SGLang metadata kernel under the target isolated configuration.Full serving gains are smaller because the optimized routine occupies only part of the decoding pipeline.
- Practical impact: Production value depends on both kernel efficiency and adoption constraints, including correctness coverage, execution-mode compatibility, maintainable code, and upstream acceptance.The SGLang and LMDeploy cases demonstrate that local operator improvements can translate into smaller but meaningful system-level gains.
- LMDeploy: More than 30% speedup is achieved by the fused LMDeploy routing kernel in isolated operator benchmarking, with end-to-end throughput gains of roughly 1.85% to 3.00%.The kernel fuses sigmoid activation, bias addition, reshape, top-k selection, and masking, and was merged into LMDeploy.
- Engram: 14.59× speedup is reported for the optimized Engram implementation in local evaluation, and its support was later merged into DLBlas.Two specialized Triton kernels fuse gate computation, RMS normalization, depthwise convolution, and residual updates.
7. Conclusion
Kernel-Smith combines stable evaluation-driven evolutionary search with evolution-oriented post-training to improve GPU kernel generation and support transfer beyond one-shot code generation. Its demonstrated scope spans KernelBench, MetaX, and real-world integrations, while future work targets broader backends and more automated, adaptive workflows.
- Conclusion: Kernel-Smith combines a stable evaluation-driven evolutionary agent with an evolution-oriented post-training recipe for high-performance GPU kernel and operator generation.The framework is evaluated across KernelBench, the MetaX backend, and real-world integrations including SGLang and LMDeploy.
- Conclusion: The results show improved search effectiveness and practical transfer beyond one-shot code generation across benchmark, heterogeneous-backend, and production-integration settings.The paper identifies reliable execution feedback and step-centric training as key ingredients of the workflow.
- Future directions: Future directions include extending backend coverage, automating more pull-request work, and developing richer tools with adaptive search strategies.
A. Example Evolution Prompt
The example evolution prompt frames kernel optimization as an iterative rewrite of a constrained Triton code block. Each candidate must compile, preserve numerical correctness, and improve performance while respecting locked interfaces and safety constraints.
- Evolution setup: The evolution step supplies archived programs and evaluator feedback so the model can propose the next candidate within an iterative search process.The prompt preserves the core structure of the system while simplifying the presentation example.
- Optimization target: The prompt targets functionally equivalent but faster Triton implementations generated from PyTorch reference modules for NVIDIA GPUs.
- Evaluation criteria: Candidates must compile, match the PyTorch reference numerically, and exceed the reference and prior generated programs in speed.
- Modification constraints: Only the Triton kernel source inside the EVOLVE-BLOCK may change; signatures, grid configuration, tensor shapes, PID logic, masks, and synchronization safety remain constrained.
- Optimization guidance: Optimization guidance emphasizes reducing global-memory access, fusing operations, tuning occupancy, increasing arithmetic intensity, and applying backend-specific techniques.Examples include BLOCK_SIZE and num_warps tuning, persistent kernels, TMA, fast_math, and advanced tiling.