PaperlayerView exact revision on arXiv

Source-linked AI summary

Making LLMs Optimize Multi-Scenario CUDA Kernels Like Experts

Yuxuan Han, Meng-Hao Guo, Zhengning Liu, Wenguang Chen, Shi-Min Hu

arXiv:2603.07169v1cs.LGstat.ML

TL;DR

Existing automated CUDA optimization methods focus mainly on LLM-oriented workloads, leaving broader multi-scenario evaluation underdeveloped. The paper introduces MSKernelBench and CUDAMaster, a profiling-guided multi-agent system with complete toolchain generation, and reports substantial gains across most operators, including about 35% over Astra.

  • Problem

    Existing benchmarks and methods focus mainly on deep-learning operators, leaving sparse, scientific, and other general-purpose GPU kernels insufficiently evaluated.

  • Method

    MSKernelBench evaluates diverse operators in FP32 and BF16, while CUDAMaster uses selectively filtered profiling data and multiple agents to generate and deploy optimized CUDA toolchains.

  • Results

    About 35%: CUDAMaster outperforms Astra, with significant speedups across most kernels and performance competitive with manually tuned libraries in several cases.

  • Takeaways & Limitations

    LLM-based agents can automate optimization of complex, low-level CUDA kernels across diverse scenarios and approximate expert-level tuning under a suitable environment and information.

  • Takeaways & Limitations

    Optimized kernels are evaluated within the benchmark’s standardized code and test logic, which agents do not alter.

Abstract

from arXiv · show

Optimizing GPU kernels manually is a challenging and time-consuming task. With the rapid development of LLMs, automated GPU kernel optimization is gradually becoming a tangible reality. However, current LLM-driven automated optimization methods narrowly focus on machine learning applications, such as PyTorch operator optimization, while overlooking broader domains like sparse matrix operations in scientific computing. Extending to these broader applications brings new challenges for the benchmark and algorithm. Therefore, developing a general-purpose automated kernel optimization method becomes our primary focus. In this paper, we address the absence of systematic evaluation for multi-scenario settings by introducing MSKernelBench, which spans multiple scenarios, including fundamental algebraic operations, common LLM kernels, sparse matrix operators, and scientific computing routines, each supporting both FP32 and BF16 precision. Building on this benchmark, we introduce CUDAMaster, a multi-agent, hardware-aware system for kernel optimization that leverages profiling information and automatically constructs the full compilation and execution toolchain. Experimental results demonstrate that CUDAMaster achieves significant speedups across most operators, outperforming Astra by about 35%. In several cases, its performance matches or surpasses that of highly optimized, closed-source libraries such as cuBLAS. A demo showcasing the original and optimized code for each operator is available at https://hanyx2021.github.io/MSKernelBenchDemo/.

1. Introduction

Existing LLM-based CUDA optimization research focuses mainly on deep-learning operators, leaving broader GPU workloads insufficiently evaluated. This paper introduces a multi-scenario benchmark and a profiling-guided multi-agent framework, reporting substantial speedups across diverse kernels.

  • Motivation: Existing benchmarks largely target PyTorch and LLM operators, overlooking sparse matrix and scientific-computing kernels.These workloads involve broader execution patterns, including irregular memory access.
  • Motivation: Multi-scenario CUDA optimization spans distinct patterns across sparse linear algebra, scientific simulations, and other domains.Hand-tuned libraries offer peak performance but require substantial engineering per operator and hardware generation.
  • Contributions: MSKernelBench covers foundational algebraic, LLM, sparse matrix, and scientific-computing operators in FP32 and BF16.The benchmark is designed to evaluate optimization capabilities without framework dependencies.
  • Contributions: CUDAMaster uses selectively filtered hardware profiling data and multiple agents to generate optimized kernels with complete compilation and execution toolchains.Its task process is designed to address performance bottlenecks while preserving operator correctness.
  • Results: About 35%: the method outperforms Astra, while sometimes matching or exceeding closed-source libraries such as cuBLAS.The reported gains occur across most operators, with several cases reaching closed-source-library performance.

2. Related Work

Prior CUDA kernel optimization uses either high-level compiler abstractions or manual and vendor-tuned implementations. Recent LLM-based systems increasingly use multi-agent, evolutionary, and training-based approaches to generate and optimize kernels.

  • Optimization paradigms: High-level approaches such as TVM and Triton use compiler abstractions and auto-tuning to improve development efficiency.Manual approaches instead rely on vendor libraries or expert-oriented template libraries.
  • Optimization paradigms: Both compiler-based and manual approaches require substantial hardware expertise and make operators hardware-dependent engineering challenges.
  • Benchmarks: KernelBench evaluates 250 PyTorch-based workloads with the fast_p metric, while later benchmarks extend robustness or backend verification.Robust-kbench varies scenarios and data shapes; MultiKernelBench adds multiple backend platforms.
  • LLM-based optimization: Recent systems combine multi-agent or evolutionary optimization, profiler feedback, and specialized agents to generate correct, high-performance CUDA kernels.Training-based methods separately learn optimization strategies or target specific kernel families.

3. MSKernelBench: A multi-scenario CUDA operator optimization benchmark

MSKernelBench broadens CUDA optimization evaluation beyond LLM-centric workloads by covering diverse HPC scenarios and varying computational scales. Its pure-C, unified-interface design and workload-weighted scoring support portable, rigorous comparisons.

  • Design rationale: Current CUDA tuning benchmarks focus mainly on LLM operators, overlooking HPC tasks with distinct memory, parallelism, and precision requirements.Single-size evaluation also fails to capture scaling across computational loads and hardware-utilization regimes.
  • Design rationale: MSKernelBench introduces broader evaluation of operator-tuning methods through a multi-scenario benchmark summarized against KernelBench.
  • Benchmark composition: The benchmark contains 50 tasks collected from authoritative NVIDIA sources, CUDA libraries, KernelBench, and LeetGPU.
  • Design rationale: Pure C provides portability, low-level control, library integration, and fine-grained control over memory access and parallelism.These properties support optimization of irregular computations such as sparse matrix operations.
  • Implementation: Each of the 50 tasks has separate FP32 and BF16 implementations accessed through a consistent C wrapper and common test-suite template.This separates kernel optimization from the testing harness.
  • Evaluation metrics: The evaluation measures speedup against a ground-truth baseline after 3 warm-up runs and 50 timed executions per data size.
  • Evaluation metrics: The overall performance score is a complexity-weighted average of measured end-to-end speedups across data sizes.Weights are proportional to the baseline implementation’s theoretical computational complexity.
  • Evaluation metrics: Complexity weighting gives larger inputs greater influence and amplifies advantages from reduced runtime complexity.The metric is intended to compare optimizations that scale differently with problem size.

4. CUDAMaster: A Multi-agent System Works like Experts

CUDAMaster automates iterative GPU-kernel tuning with specialized agents, hardware-bottleneck filtering, profiling, compilation, execution, and correctness checks. Its pipeline preserves the benchmark’s standardized testing environment while updating the best valid kernel across optimization rounds.

  • Motivation: Manual optimization is slowed by data triage and context switching among correctness debugging, performance analysis, and code implementation.CUDAMaster addresses these workflow bottlenecks by filtering profiling data and assigning stages to specialized agents.
  • Hardware-aware profiling: The hardware-analysis filter classifies each kernel as compute-, memory-latency-, or memory-bandwidth-bound and retains only bottleneck-relevant metrics.A unified mapping table defines classification criteria and associated metrics, reducing redundant profiling context for downstream agents.
  • System architecture: CUDAMaster uses four specialized agents to automate kernel tuning across R iteration rounds and up to D debugging rounds.The framework is governed by Algorithm 1 and separates planning, coding, compilation, and debugging responsibilities.
  • Optimization pipeline: The pipeline begins with baseline execution and profiling, then iteratively plans, codes, compiles, executes, validates, and retains only valid performance improvements.Invalid kernels trigger a debugging subprocess limited to D rounds; valid kernels replace the incumbent when their performance improves.
  • Evaluation protocol: All optimized kernels remain anchored to the benchmark’s standardized code and test logic, enabling consistent and fair evaluation.The agents do not alter the core test logic during optimization.

5. Experiment

The experiments evaluate CUDAMaster across 100 precision-specific kernel tasks using macro- and micro-level performance metrics. Results show stronger cumulative success for o4-mini, competitive single-operator performance, and benefits from iterative debugging and filtered profiling.

  • Experimental Setup: 100 kernel optimization tasks are evaluated across 50 operators implemented in FP32 and BF16.
  • Metrics: The evaluation combines cumulative success across tasks with per-operator speedup against authoritative libraries or research baselines.
  • Cumulative Success: 100% vs. 95% success is achieved at τ=0, 94% vs. 80% at τ=1, and 60% vs. 49% at τ=2 for o4-mini versus DeepSeek-V3.2.BF16 slightly benefits o4-mini at moderate-to-high thresholds, while FP32 is more stable at stringent targets.
  • Single-Operator Speedup: Up to 1.8× speedup over cuDNN in 2D Convolution and cuBLAS in Dot Product is reported, while both models outperform cuSPARSE in SpMV CSR.Against Astra, o4-mini leads by around 35% on RMSNorm and matches or surpasses it on the other fused operators.
  • Ablations: 94% success at τ=1 is achieved by the Full configuration, versus 77% with No Debug and 74% with Single Iteration.At τ≥8, Full maintains 25% success compared with 17–19% for ablated variants.

6. Conclusion

The paper introduces MSKernelBench as a diverse testbed and CUDAMaster as an end-to-end multi-agent CUDA optimization framework. Across diverse scenarios, the system achieves significant speedups and competitive performance with manually tuned libraries.

  • MSKernelBench spans multiple domains and numerical precisions to provide a rigorous testbed for GPU code optimization.CUDAMaster generates, debugs, and deploys optimized CUDA code together with its compilation and execution toolchain.
  • CUDAMaster produces significant speedups across the benchmark and competitive performance with manually tuned libraries in several cases.The conclusion frames LLM-based agents as capable of automating optimization of complex, low-level CUDA kernels across diverse scenarios.

A.1. Full Metrics

The appendix identifies the hardware performance counters collected with NVIDIA Nsight Compute and organizes the filtered metrics by profiling section.

  • NVIDIA Nsight Compute v2024.1 supplies fine-grained hardware performance counters for the framework.
  • Table 6 organizes the filtered hardware performance metrics used for bottleneck analysis by profiling section.

A.2. Bottleneck Classification Logic

CUDAMaster classifies kernel bottlenecks using throughput metrics and data-driven thresholds calibrated with Otsu’s method. The resulting rules distinguish compute, memory-latency, and memory-bandwidth limitations.

  • Metric-Based Classification: Duration identifies the most time-consuming bottleneck kernel, whose baseline execution yields three throughput metrics for classification.The metrics are Compute (SM) Throughput, DRAM Throughput, and Memory Throughput.
  • Threshold Calibration: Otsu’s method selects thresholds that maximize separation between high- and low-utilization distributions for each throughput metric.This calibration replaces purely heuristic thresholds with objective, data-driven criteria.
  • Threshold Calibration: Thresholds near 30% classify high Compute (SM) Throughput as computational load and low DRAM or Memory Throughput as memory underutilization.
  • Classification Rules: Compute Bound requires Compute (SM) Throughput > 30%.This category indicates that computational units are the primary active bottleneck.
  • Classification Rules: Memory Latency Bound requires DRAM Throughput < 30% and Memory Throughput < 30%, with low cache hit rates indicating long data-fetch stalls.
  • Classification Rules: Memory Bandwidth Bound is the default category when the preceding criteria are unmet, especially when DRAM Throughput is high.It represents performance limited by a saturated memory interface and potentially complex access patterns.
  • Classification Rules: The three-way heuristic directly informs the hardware-aware optimization strategy.

A.3. Interpretation of Filtered Metrics per Bottleneck Type

The hardware analysis filter narrows profiling data according to the classified bottleneck, giving the planning agent actionable signals for targeted optimization.

  • The filter selects metrics most relevant to diagnosing the root cause of each classified bottleneck.This reduces profiling noise and focuses subsequent planning on actionable hardware signals.
  • Compute-bound kernels are assessed through SM throughput, issue-slot activity, instruction throughput, and SM utilization.Compute (SM) Throughput is the primary performance scorecard, while low Issue Slots Busy can indicate front-end inefficiencies.
  • Memory-latency-bound kernels are diagnosed using cache-hit rates, cycles per instruction, elapsed IPC, and memory activity.Low cache-hit rates identify poor locality and can motivate tiling or shared-memory strategies.
  • Memory-bandwidth-bound kernels are evaluated with DRAM throughput, memory throughput, maximum bandwidth, and memory-pipeline utilization.DRAM Throughput approaching theoretical Max Bandwidth indicates saturation and directs optimization toward reducing or improving data movement.
  • Contextualized metric filtering enables the planning agent to generate more targeted optimization proposals.

A.4. Bottleneck Distribution and Task-Type Correlation

MSKernelBench contains 50 tasks distributed across compute, memory-latency, and memory-bandwidth bottlenecks, whose task types correspond to distinct computational and memory-access patterns.

  • 50 benchmark tasks comprise 11 Compute Bound, 24 Memory Latency Bound, and 15 Memory Bandwidth Bound tasks.Figure 7 presents the distribution across the three bottleneck categories.
  • Compute-bound tasks include dense linear algebra, stencil computation, and core LLM operators with high arithmetic intensity and predictable memory access.
  • Memory-latency-bound tasks include normalization, loss, and activation functions characterized by low reuse, scattered accesses, and low cache efficiency.
  • Memory-bandwidth-bound tasks include sparse matrix and tensor-processing routines that move large datasets and can saturate memory bandwidth.
  • The correlation between task types and bottleneck categories supports the distinction between their underlying hardware performance limitations.

A.5. Optimization-Induced Transformation of Hardware Bottleneck Profiles

The analysis evaluates optimization through bottleneck-distribution shifts and within-category hardware-metric changes, showing reduced memory latency pressure and improved utilization.

  • Evaluation approach: The evaluation combines macroscopic bottleneck-classification shifts with microscopic hardware-metric improvements.This provides both a high-level effectiveness view and granular evidence about performance changes.
  • Bottleneck transformations: 6 memory-bound tasks became compute-bound, indicating reduced memory pressure and greater computational intensity.
  • Bottleneck transformations: 14 tasks shifted from memory latency to memory bandwidth, moving toward a more predictable and optimizable bottleneck.
  • Bottleneck transformations: Memory latency-bound tasks decreased by 67%, from 24 to 8.This substantially reduced the most performance-critical bottleneck identified in the analysis.
  • Bottleneck transformations: Compute-bound tasks increased by 45% and memory-bandwidth-bound tasks by 73%.These shifts indicate a better balance with the targeted GPU resource categories.
  • Metric improvements: Compute-bound kernels improved Issue Slots Busy by 16.8% and Executed Ipc Active by 15.4%.
  • Metric improvements: Memory-latency-bound kernels improved Mem Busy by 1312.7% and Executed Ipc Elapsed by 889.4%.
  • Metric improvements: Memory-bandwidth-bound kernels improved DRAM Throughput by 537.3% and Mem Pipes Busy by 98.9%.

C. Detailed content of MSKernelBench

Table 7 presents the detailed content of each MSKernelBench task across multiple continued table sections.

  • Table 7 is identified as the detailed content listing for each task in MSKernelBench.
  • The detailed task listing continues across several table segments.
  • Additional continuation segments are labeled as part of Table 7’s detailed task content.
Loading 2603.07169v1…