PaperlayerView exact revision on arXiv

Source-linked AI summary

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma, An Zou

arXiv:2603.02236v1cs.LGcs.AI

TL;DR

Existing benchmarks largely assess code-to-code CUDA translation and provide limited performance evidence for the broader text-to-CUDA task. CUDABench introduces a multidimensional dataset, execution-based verification, and a roofline-based score; evaluations show high compilation success alongside weaker functional correctness and incomplete hardware utilization.

  • Problem

    Existing benchmarks focus on translating high-level code into CUDA, leaving text-to-CUDA generation and hardware-independent performance assessment insufficiently evaluated.

  • Method

    CUDABench combines CUDABench-Set's Breadth-Depth-Difficulty coverage with execution-based verification and the roofline-based, hardware-independent CUDABench-Score.

  • Results

    Evaluated LLMs achieve high compilation success but show a notable gap in functional correctness, limited domain-specific knowledge, and incomplete use of GPU hardware resources.

  • Takeaways & Limitations

    CUDABench establishes a foundation for systematically studying CUDA kernel generation beyond syntax and code translation.

Abstract

from arXiv · show

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.

1. Introduction

CUDABench addresses the limited evaluation of general text-to-CUDA generation, which existing code-translation benchmarks do not capture. It combines broad task coverage with verification and hardware-independent performance assessment to evaluate LLM-generated CUDA kernels.

  • Motivation: Existing benchmarks primarily translate high-level code into CUDA, whereas text-to-CUDA generation requires producing kernels from computational intent described in text.This makes text-to-CUDA a more general evaluation setting than code-to-code translation.
  • Motivation: Correct outputs alone are insufficient for CUDA evaluation because kernel performance depends on balancing computational throughput, memory bandwidth, and hardware architecture.Functionally correct kernels may still perform substantially worse than optimized implementations.
  • CUDABench: CUDABench targets text-to-CUDA generation across multiple application domains and incorporates hardware-independent metrics reflecting practical GPU performance.The benchmark is designed to address both task coverage and performance comparability.
  • Contributions: CUDABench-Set spans Breadth-Depth-Difficulty through multiple domains, input scales, and prompt-detail levels.Its dimensions are intended to vary the scope, scale, and difficulty of text-to-CUDA tasks.
  • Contributions: The Generative Verification Pipeline evaluates compilation and functional correctness through execution-based verification, while CUDABench-Score measures generated-kernel performance beyond execution time.Together, these components assess correctness and performance rather than compilation alone.
  • Contributions: Extensive evaluation of state-of-the-art LLMs reveals strengths, limitations, and characteristic behaviors in CUDA kernel generation.The evaluation is presented as a benchmark-wide analysis of current models.

2. Related Works

Prior benchmarks evaluate CUDA generation mainly through PyTorch-to-CUDA translation or limited natural-language tasks, and often lack reference implementations or robust performance assessment. CUDABench broadens task scope and evaluates compilation, functional correctness, and hardware-independent performance.

  • Existing Benchmarks: KernelBench jointly measures functional correctness and performance but is limited to PyTorch-to-CUDA translation and largely PyTorch machine-learning workloads.It also does not provide reference CUDA implementations.
  • Existing Benchmarks: ComputeEval broadens CUDA kernel generation to natural-language tasks but lacks reference implementations, uses relatively small-scale test cases, and omits performance evaluation.These limitations constrain assessment of practical performance and real-world engineering relevance.
  • Performance Evaluation: Execution time alone is hardware-sensitive because it depends on GPU specifications, memory bandwidth, system load, and other factors.This makes direct performance comparisons across platforms and settings challenging.
  • CUDABench: CUDABench evaluates text-to-CUDA generation across multiple application domains using compilation correctness, functional correctness, and roofline-based hardware-independent CUDABench-Score.The benchmark is presented as a response to the scope and measurement limitations of prior work.

3. Text-to-CUDA Dataset: CUDABench-Set

CUDABench-Set is a text-to-CUDA dataset organized across application breadth, input-scale depth, and prompt difficulty. Its tasks use verified references and scale-specific validation to evaluate correctness and performance across diverse GPU workloads.

  • Dataset Construction: CUDABench-Set contains CUDA-kernel tasks specified by textual prompts describing kernels, data types, hardware, and related requirements.Each task is supported by a data generator and validator using randomized inputs and reference outputs.
  • Breadth: The dataset covers six GPU-computing domains, including linear algebra, deep-learning operators, computer vision, data analytics, signal processing, and scientific simulation and finance.These domains include workloads such as GEMM, normalization, sorting, filtering, PDE solving, and Black-Scholes pricing.
  • Breadth: Tasks are derived from open-source CUDA codebases, including deep-learning operator libraries, NVIDIA CUDA Samples, and production-level GPU repositories.The task distribution across domains is shown in Figure 2a.
  • Depth: Five input-scale levels—Tiny, Small, Medium, Large, and Huge—range from a few kilobytes to over 1 GB to stress GPU compute units across production-like sizes.The Haar Wavelet Transform example scales signal lengths from 512 to 8,192.
  • Depth: Each input size has a dedicated reference kernel, data generator, and validator because implementation and validation criteria vary across scales.This alignment adapts reference outputs and validation logic to each input size.
  • Difficulty: Three prompt difficulty levels progressively remove implementation and contextual details, from guided CUDA mapping to algorithmic specification and task-name-only concept retrieval.The hierarchy evaluates capabilities ranging from syntax generation to autonomous domain-knowledge retrieval.
  • Dataset Construction: The dataset comprises 1,500 prompts from 500 tasks paired with three difficulty levels, with each task equipped with a verified reference CUDA kernel.These references support the accompanying data generator and validator.

4. Text-to-CUDA Metrics: CUDABench-Score

CUDABench combines strict correctness filtering with roofline-based performance metrics to evaluate generated CUDA kernels. Its unified CUDABench-Score integrates compilation, functional consistency, and bottleneck-aware performance across an automated verification pipeline.

  • Correctness Metrics: Generated kernels must compile successfully and match reference outputs before performance benchmarking.The pipeline applies compilation and execution-based functional checks, and only valid kernels proceed to performance evaluation.
  • Roofline-Based Performance: The roofline model characterizes kernels using execution time, FLOPs, data movement, and arithmetic intensity to estimate attainable performance.Attainable performance is bounded by the minimum of peak floating-point throughput and peak memory bandwidth multiplied by arithmetic intensity.
  • Roofline-Based Performance: Performance-Score is the ratio of achieved GFLOPs/sec to attainable GFLOPs/sec, enabling normalized comparisons across different bottleneck regimes.The metric compares achieved performance with the theoretical hardware limit for each kernel.
  • Empirical Comparison: On MSE kernels, optimized implementations reached Performance-Scores of 78.6% on A40 and 74.8% on RTX 4090, versus 0.4% and 0.3% for original kernels.The optimized kernels achieved 295.2 and 554.9 GFLOPs/sec, while original kernels achieved 115.4 and 184.5 GFLOPs/sec on the respective GPUs.
  • Roofline-Based Performance: Performance-Score measures memory-bandwidth utilization for memory-bound kernels and compute utilization for compute-bound kernels.This adapts the metric to the kernel’s dominant performance bottleneck.
  • Unified Benchmark Score: CUDABench-Score integrates compilation correctness, functional consistency, and Performance-Score into a scalar measure of reliable, high-performance kernel generation.The verification pipeline automates test-data preparation, compilation, functional verification, and performance evaluation.

5. Experiment

CUDABench evaluates recent LLMs across difficulty levels using compilation, functional correctness, and performance metrics. Results show declining correctness with task difficulty, substantial semantic failures despite compilation success, domain-specific weaknesses, and consistently low hardware utilization.

  • Overall results: Claude 4.5 Sonnet reaches 99.8% compilation and 85.8% functional accuracy at Level 1, while GPT-5.2 leads CUDABench-Score at 40.9%.Claude leads correctness, whereas GPT-5.2 maintains the performance-score advantage across higher difficulty levels.
  • Difficulty effects: GPT-5.2’s functional accuracy falls from 79.8% at Level 1 to 60.8% at Level 3 under Pass@1.The decline indicates increasing difficulty as kernel complexity grows.
  • Difficulty effects: Pass@3 improves over Pass@1, indicating that multiple sampling attempts can produce correct CUDA kernels more often.The paper identifies prompting, sampling strategies, and domain-specific training as possible routes for improvement.
  • Correctness analysis: Functional errors dominate compilation failures, reaching approximately 47% for DeepSeek-V3.2 at Level 3 versus 3.6% compilation faults.Claude 4.5 Sonnet can achieve 100% compilation across difficulty levels while still showing declining functional accuracy.
  • Domain-specific performance: Claude 4.5 Sonnet reaches an 85.0% failure rate in Scientific Simulation & Finance on Level 3, compared with GPT-5.2’s 8.0% rate in Linear Algebra.The disparity reflects weaker performance on complex, niche algorithms than on foundational mathematical tasks.
  • Performance analysis: GPT-5.2, Claude 4.5 Sonnet, and Gemini 3 Flash achieve nearly identical CUDABench-Scores of 40.9%, 40.2%, and 40.1%.At Level 1, these scores leave approximately 60% of GPU computational resources unexploited.
  • Performance analysis: Qwen3 Max scores 28.0% on the RTX 4090 and 32.1% on the A40 at Level 1, supporting hardware normalization by CUDABench-Score.The supplementary comparison tests whether the metric remains stable across substantially different GPU specifications.

6. Conclusion

CUDABench introduces a hardware-independent benchmark for text-to-CUDA generation, combining breadth, depth, and difficulty with a roofline-based performance metric. Its evaluation finds high compilation success but persistent functional, domain-knowledge, and GPU-utilization limitations.

  • Conclusion: CUDABench combines Breadth-Depth-Difficulty evaluation with the roofline-based CUDABench-Score for systematic text-to-CUDA assessment.The benchmark evaluates generated kernels beyond compilation success alone.
  • Conclusion: State-of-the-art LLMs compile CUDA kernels successfully but struggle with functional correctness, domain-specific knowledge, and full GPU-resource utilization.These findings establish CUDABench as a foundation for research on CUDA kernel generation.

Example

The example prompt specifies a CUDA implementation of the Haar wavelet transform for an 8192-value float32 signal, partitioned across eight blocks with recursive pairwise operations. It also defines output placement for detail coefficients and block-wise approximation coefficients.

  • The task applies the Haar wavelet transform to an 8192-element float32 signal.
  • Each of eight blocks processes 1024 elements using 512 threads.
  • Within each block, the kernel recursively applies pairwise operations to generate detail and approximation coefficients.
  • All detail coefficients are stored in an 8192-element output tensor, with level-1 details occupying its second half.
  • The final approximation coefficient from each block is stored in an 8-element output tensor.
Loading 2603.02236v1…