PaperlayerView exact revision on arXiv

Source-linked AI summary

CUDA-Harness: Harnessing Agentic CUDA Kernel Generation and Optimization from Natural Language

Qi Fan, An Zou, Yehan Ma

arXiv:2609.00058v1cs.CLcs.AIcs.MAcs.PLcs.SE

TL;DR

High-performance CUDA kernel development remains difficult because Text2CUDA starts from underspecified natural language and lacks the detailed semantics and test data available in Torch2CUDA. CUDA-Harness uses structured generation, synthesized progressive verification, and correctness-prioritized evolution, with experiments showing effectiveness and generalization across LLMs, hardware platforms, and C-to-CUDA transpilation.

  • Problem

    Text2CUDA must derive high-level input semantics while implementing and validating low-level kernels from underspecified natural-language prompts without available test data.

  • Method

    CUDA-Harness combines Intermediate-Structured Generation, Synthesis-Based Verification, and Feedback-Adaptive Evolution for kernel generation, verification, and optimization.

  • Results

    CUDA-Harness demonstrates effectiveness, with evaluations illustrating generalization across LLMs, hardware platforms, and C-to-CUDA transpilation.

  • Takeaways & Limitations

    Natural language can serve as a practical interface for developing correct and high-performance CUDA kernels within the evaluated scope.

  • Takeaways & Limitations

    Validation can produce false positives from prompt misinterpretation, omitted post-processing, and incorrect data-type loading in a benchmark template.

Abstract

from arXiv · show

Developing high-performance CUDA kernels demands specialized knowledge in algorithm implementation, correctness validation, and hardware-aware parallel optimization, creating a substantial expertise barrier and making generating CUDA kernels directly from natural language (Text2CUDA) essential. Meanwhile, the general-purpose code generation capability of Large Language Models (LLMs) prompts a series of works exploring LLM-based CUDA kernel generation. They mainly focus on transpilation from high-level frameworks such as PyTorch to CUDA (Torch2CUDA) rather than Text2CUDA, where models must understand the high-level input semantics and handle low-level kernel implementation and validation. Additionally, these methods are vulnerable to reward hacking due to reliance on predefined test inputs. In this paper, we propose CUDA-Harness, a framework for harnessing agentic CUDA kernel generation and optimization from natural language. Specifically, we introduce Intermediate-Structured Generation to connect high-level semantic understanding with low-level kernel generation. To dilute reward hacking in Text2CUDA, we construct Synthesis-Based Verification to provide isolated test data and progressive validation. Furthermore, we propose Feedback-Adaptive Evolution, a kernel evolution strategy that prioritizes correctness while optimizing performance. Finally, through extensive experiments, we demonstrate the effectiveness of CUDA-Harness, with further evaluations illustrating generalization across LLMs, hardware platforms, and to C-to-CUDA transpilation.

1 INTRODUCTION

Text2CUDA aims to make high-performance CUDA kernel development accessible from natural language, but requires semantic understanding, implementation, validation, and optimization from underspecified inputs. CUDA-Harness addresses this challenge with structured generation, synthesized verification, correctness-prioritized evolution, and evaluations across models, hardware, and transpilation settings.

  • High-performance CUDA development requires algorithmic, testing, hardware, and parallel-optimization expertise, creating a substantial accessibility barrier.
  • Text2CUDA differs from Torch2CUDA because it begins with underspecified natural language rather than algorithmically detailed PyTorch code.
  • Existing LLM-based CUDA approaches rely on training data or agentic refinement, while Text2CUDA must additionally resolve input semantics and low-level implementation.
  • CUDA-Harness introduces Intermediate-Structured Generation to bridge high-level semantic understanding and low-level kernel synthesis.
  • Synthesis-Based Verification supplies isolated test-data synthesis and progressive validation to dilute reward hacking in Text2CUDA.
  • Feedback-Adaptive Evolution prioritizes correctness while optimizing kernel performance, and experiments evaluate effectiveness and generalization across LLMs, hardware, and C-to-CUDA transpilation.

2 RELATED WORK

Prior LLM-based CUDA work includes training-based generation and agent-based test-time optimization, but it primarily targets transpilation rather than general Text2CUDA. Existing Text2CUDA efforts address iterative improvement and benchmarking, while remaining constrained by current kernel-generation limitations.

  • LLM-based CUDA kernel generation methods broadly divide into training-based and agent-based approaches.
  • Training-based methods use supervised fine-tuning or reinforcement learning to improve CUDA generation, but require scarce, expensive high-quality kernels and optimization trajectories.
  • Agent-based methods use multi-agent collaboration, iterative refinement, profiling, and search to generate and optimize kernels at test time.
  • Existing work primarily targets Torch2CUDA-style transpilation, whereas Text2CUDA begins from more general natural-language intents.
  • CUDA-LLM iteratively improves Text2CUDA kernels under validation, while CUDABench evaluates Text2CUDA capabilities across diverse domains.

3 OVERVIEW

CUDA-Harness treats the self-contained CUDA program as the unit that supplies compilation, execution, and validation context. Its overview connects underspecified natural-language intents to structured planning, test-data synthesis, and kernel implementation challenges.

  • 3.1 ANATOMY OF SELF-CONTAINED CUDA: A self-contained CUDA program provides the complete context needed to compile, execute, and validate a kernel.
  • 3.1 ANATOMY OF SELF-CONTAINED CUDA: Its device-side kernels perform memory access and parallel computation, while host-side code prepares inputs, manages memory, and launches kernels.
  • 3.1 ANATOMY OF SELF-CONTAINED CUDA: Validation and performance measurement require well-prepared test inputs because problem scale and input dimensions affect runtime bottlenecks.
  • 3.2 FROM ANATOMY TO CUDA-HARNESS: Text2CUDA lacks the algorithmic detail and test data naturally available in Torch2CUDA inputs, making semantic completion and validation necessary.
  • 3.2 FROM ANATOMY TO CUDA-HARNESS: The framework identifies bridging intent with implementation and synthesizing test data without reward hacking as central Text2CUDA challenges.
  • 3.2 FROM ANATOMY TO CUDA-HARNESS: Intermediate-Structured Generation first produces an implementation manifest, then instantiates a CUDA scaffold for each planned kernel implementation.

4 CUDA-HARNESS

This section presents the components of CUDA-Harness.

  • CUDA-Harness is presented as a framework composed of multiple components.
  • The section introduces the components of CUDA-Harness as the focus of its presentation.
  • CUDA-Harness components are presented as the subject of the section.

4.1 INTERMEDIATE-STRUCTURED GENERATION

Intermediate-Structured Generation separates semantic planning from CUDA implementation through a per-kernel manifest and a self-contained scaffold. The scaffold centralizes launch-related host operations and restricts access to prepared test inputs.

  • Kernel Implementation Manifest: The kernel implementation manifest is a standalone per-kernel specification containing implementation, reference-code, and test-shape information.It records kernel descriptions, a language-agnostic reference snippet, and test input/output shapes and data types.
  • Kernel Implementation Manifest: The agent produces the manifest before writing concrete kernel code, allowing it to resolve underspecified high-level semantics separately from implementation.This division lets the agent focus on semantic understanding and missing details before coding.
  • Self-Contained CUDA Scaffold: The self-contained CUDA scaffold is a template with placeholders for the program components that the agent fills from a manifest entry.It provides the structure for implementing the complete self-contained CUDA program.
  • Self-Contained CUDA Scaffold: Kernel launch operations are aggregated into a dedicated launcher function, including memory management, data transfer, and launch configuration.This lets the agent concentrate on the kernel and launcher rather than scattered host-side details.
  • Self-Contained CUDA Scaffold: The scaffold prohibits dummy inputs and reads prepared test inputs through command-line paths, keeping their details hidden during implementation.The calling convention is emitted at compile time, reducing opportunities to tailor the kernel to specific test data.

4.2 SYNTHESIS-BASED VERIFICATION

Synthesis-Based Verification separates test-data synthesis from kernel generation and validates kernels progressively. Its protocol uses isolated data and a fail-fast sequence from compilation to correctness and then performance.

  • Decoupled Test Data Synthesis: Synthesis-Based Verification decouples test-data synthesis from kernel generation to provide isolated inputs and reference outputs for validation.Its purpose is to provide a trustworthy correctness signal when benchmark test data is unavailable.
  • Decoupled Test Data Synthesis: Shared contexts can let an agent adapt test inputs and reference outputs to its kernel, creating reward-hacking risk.Separate contexts are used to avoid this cross-contamination.
  • Decoupled Test Data Synthesis: When inputs concentrate on small magnitudes, absolute tolerance can dominate relative tolerance and allow numerically incorrect kernels to pass.The paper recommends broad numeric ranges, while noting that larger magnitudes can increase false negatives.
  • Progressive Multi-Stage Validation: Validation proceeds in fail-fast order: compilation, functional correctness, and runtime performance.Later performance checks are run only after earlier compilation and correctness checks succeed.
  • Progressive Multi-Stage Validation: MCP-based tools provide uniform compilation, functional-validation, and profiling interfaces while execution remains tied to the deployed hardware environment.This design supports validation across hardware environments.
  • Progressive Multi-Stage Validation: The agent invokes compilation first, functional validation after successful compilation, and profiling only after correctness passes.The functional tool uses the scaffold’s calling convention, and profiling measures runtime with tools such as Nsight Systems and Nsight Compute.

4.3 FEEDBACK-ADAPTIVE EVOLUTION

Feedback-Adaptive Evolution combines correctness-first refinement with reward-driven optimization to evolve CUDA kernels while preserving correctness. It uses validation-derived rewards and accumulated optimization insights to guide iterative test-time improvement.

  • Feedback-Adaptive Evolution combines correctness-first refinement and reward-driven optimization for test-time kernel evolution.
  • The agent prioritizes compilation and functional correctness before applying performance optimization, preventing refinement from remaining trapped in faulty implementations.Optimization on a functionally flawed kernel can preserve the same faulty pattern.
  • Validation failures trigger targeted repair: compilation errors receive error-guided fixes, while functional failures cause retreat to a more conservative implementation.
  • Compilation, functional validation, and profiling provide measurable rewards that drive optimization through RLVR.The reward is derived from results on the target hardware rather than estimated.
  • The agent summarizes implementation differences and reward changes as optimization insights, which accumulate across iterative rounds to guide subsequent kernel generation.Each round generates multiple kernels and retains the accumulated insights for later optimization.

5 EVALUATION

The evaluation uses CUDABench with a fixed LLM and hardware setup to isolate the harness effect. It measures compilation success, functional correctness, and runtime performance under iterative kernel optimization.

  • The evaluation uses CUDABench prompts spanning three difficulty levels that progressively remove prompt details.
  • Seed2.0 Lite is fixed across evaluations, with generation performed once per kernel to isolate the harness from LLM variation.
  • Experiments run on an NVIDIA A40 GPU with Ampere architecture and sm 80 compute capability.
  • Optimization uses three rounds with three generated kernels per round, while Nsight Systems computes performance from average latency over multiple runs.The runtime metric is based on repeated execution during performance measurement.
  • Evaluation reports compilation success and functional correctness rates, plus RScore defined as 1/execution time in milliseconds, with incorrect kernels assigned zero.RScore is an average and is not a percentage.

5.2 MAIN RESULTS

CUDA-Harness outperforms the baseline across difficulty levels on compilation, functional correctness, and performance, with larger gains as prompts become less informative. Ablations show contributions from intermediate structuring, reward-driven optimization, and correctness-first refinement, while synthesized validation largely agrees with benchmark validation.

  • 5.2.1 COMPARISON WITH BASELINES: CUDA-Harness outperforms the baseline across all difficulty levels in compilation success, functional correctness, and performance score.
  • 5.2.1 COMPARISON WITH BASELINES: As prompt difficulty increases, the performance gap grows because Intermediate-Structured Generation completes underspecified details before low-level implementation.
  • 5.2.2 ABLATION STUDY: CUDA-ISG outperforms the baseline, supporting the effectiveness and necessity of the proposed intermediate blueprints.
  • 5.2.2 ABLATION STUDY: Reward-driven optimization improves compilation success and performance, while correctness-first refinement further improves functional correctness during optimization.
  • 5.2.3 EFFICACY OF THE SYNTHESIS-BASED VERIFICATION: 84.4% accuracy, 89.4% precision, and 91.4% recall measure agreement between synthesized-test validation and benchmark validation.
  • 5.2.3 EFFICACY OF THE SYNTHESIS-BASED VERIFICATION: 8.7% of cases are false positives, arising from prompt misinterpretation, omitted benchmark-specific post-processing, or a data-type defect in the benchmark template.

5.3 GENERALIZATION ANALYSIS

CUDA-Harness generalizes across underlying LLMs, GPU hardware platforms, and C-to-CUDA transpilation. Across these settings, it consistently improves over the corresponding baseline in correctness-related and performance measures.

  • GENERALIZATION ACROSS LLMS: Across all three evaluated LLMs, CUDA-Harness outperforms the corresponding baseline in compilation success rate, functional correctness rate, and runtime performance score.The evaluation uses Seed2.0 Lite, DeepSeek-V3.2, and GLM-5.1 on an NVIDIA A40 GPU.
  • GENERALIZATION ACROSS HARDWARE: Across three GPUs with distinct architectures and compute capabilities, CUDA-Harness consistently outperforms the baseline on performance score.Compilation success and functional correctness rates remain close across platforms, while performance varies with hardware capability.
  • C-TO-CUDA TRANSPILATION: CUDA-Harness outperforms the baseline for C-to-CUDA transpilation even with a relatively small evolution budget.The C-to-CUDA evaluation uses one optimization round and one generated kernel per round.
  • C-TO-CUDA TRANSPILATION: Intermediate-Structured Generation supports C-to-CUDA generalization because it decouples input understanding from CUDA implementation.The method is not tied to natural-language input and helps the agent capture C code before generating the CUDA kernel.

6 CONCLUSION

CUDA-Harness addresses Text2CUDA by structuring semantic understanding, synthesizing isolated validation data, and evolving kernels with correctness prioritized before performance. Experiments support its effectiveness and generalization across LLMs, hardware platforms, and C-to-CUDA transpilation.

  • CONCLUSION: CUDA-Harness bridges high-level intents and low-level CUDA implementation through Intermediate-Structured Generation.The framework introduces an intermediate representation to connect semantic understanding with kernel generation.
  • CONCLUSION: Synthesis-Based Verification provides isolated test-data synthesis and progressive validation for Text2CUDA.Its purpose is to support trustworthy correctness signals when benchmark test data is unavailable during test-time evolution.
  • CONCLUSION: Feedback-Adaptive Evolution prioritizes kernel correctness while optimizing performance during test-time evolution.The strategy uses verification-guided refinement to balance correctness and efficiency.
  • CONCLUSION: CUDA-Harness generalizes across diverse hardware platforms and extends effectively to C-to-CUDA transpilation.The conclusion reports these extensions alongside the framework’s demonstrated effectiveness.
Loading 2609.00058v1…