Improving Efficiency of GPU Kernel Optimization Agents using a Domain-Specific Language and Speed-of-Light Guidance

1 Introduction

• Low-level agents must choose optimization strategies while generating complex, architecture-specific code, and profiling gives only a local view of performance rather than remaining headroom.Each candidate requires compilation, correctness testing, and profiling, making wasted iterations costly.Source: §1 · pp. 1–2
• µCUTLASS raises the abstraction from raw code to a compact, statically validated DSL that exposes high-impact kernel choices while hiding implementation complexity.It covers operator selection, kernel configuration, epilogue fusion, and multi-stage pipelines.Source: §1 · p. 2
• SOL analysis estimates performance headroom from problem requirements and hardware specifications, then guides optimization hypotheses, resource allocation, and integrity checks.MANTIS uses the SOL signal to prioritize hypotheses and shift effort away from problems near the bound or stagnating.Source: §1 · p. 2
• The evaluation uses 59 KernelBench problems selected for modern and emerging text LLM workloads, and focuses on open-source kernel generation rather than closed-source implementations.The subset excludes pooling and legacy CV/RNN tasks.Source: §1 · p. 2
• 1.56× geomean speedup over PyTorch for GPT-5-mini with SOL-guided steering builds on µCUTLASS, which reaches 1.27× where the naive agent regresses 0.40×.These results use the same iteration budget on a 59-problem KernelBench subset.Source: §1 · pp. 2–3
• SOL-guided scheduling saves 19–43% of tokens while retaining at least 95% of geomean speedup; the best configuration reaches a 1.68× efficiency gain.SOL analysis also helps detect benchmark gaming; without integrity filtering, reported speedups would inflate by up to 1.9×.Source: §1 · p. 3

2.1 Compiler Autotuning and Scheduling Systems

• Compiler autotuning systems search constrained program or schedule spaces; Halide [Ragan-Kelley et al. (2013)], OpenTuner [Ansel et al. (2014)], TVM [Chen et al. (2018)], Ansor [Zheng et al. (2020)], and MetaSchedule [Shao et al. (2022)] advance this approach.The paper distinguishes its setting by requiring an in-context-learnable DSL for LLM agents and static rejection of invalid configurations before costly toolchain runs.Source: §2.1 · p. 3

2.2 GPU Programming DSLs and Kernel Libraries

• CUTLASS/CuTe expose rich, near-SOL kernel choices but require template-heavy code, while Triton reduces boilerplate yet still requires full-kernel synthesis and low-level reasoning.µCUTLASS trades expressiveness for reliability by exposing performance-critical CUTLASS knobs and hiding scaffolding.Source: §2.2 · p. 3

2.3 Performance Bounds

• The roofline model [Williams et al. (2009)] frames compute- versus memory-bound performance using peak hardware throughput; later work applies or extends bound analysis for LLM inference and constrained data movement.The cited extensions include LLM inference [Yuan et al. (2024)] and simulation or mapping-based bounds [Huang et al. (2024)].Source: §2.3 · p. 3

2.4 LLMs for Kernel Generation and Optimization

• Kernel optimization agents use execution feedback, planning, evolutionary search, reinforcement learning, profiling, and controller methods, while this work changes representation and uses SOL for headroom, stopping, and budget reallocation.The study also treats context management as largely orthogonal and passes forward limited information from prior attempts.Source: §2.4 · p. 4

3 µCUTLASS: A DSL for CUTLASS GPU Kernels

• µCUTLASS is designed to be learnable in context, support high-level reasoning, and retain a small set of high-impact control choices rather than expose the full target language.The design goals include common patterns such as fused epilogues and explicit pipeline boundaries.Source: §3 · p. 4
• The DSL exposes CUTLASS decisions while hiding template instantiation and scaffolding, and is not intended as a general GPU programming language.Its goal is to capture the design-space choices that drive much of performance variation while keeping the language small enough to learn in context.Source: §3 · pp. 4–5
• µCUTLASS supports kernel configuration, epilogue fusion with >>, and multi-stage pipelines composed with pipeline(...).Configuration includes operation, data types, layouts, tile shapes, scheduling policy, alignment, and pipeline stages.Source: §3 · p. 5
• Specifications are often about 10–20 lines, with explicit named choices that shift boilerplate and validity checks to the compiler.The compiler and DSL are tested with standalone examples that check kernel correctness against PyTorch references.Source: §3 · pp. 5–6
• The compiler parses DSL programs into a typed configuration IR, validates constraints, and emits CUTLASS implementations; pipeline programs can include transform stages around the main kernel.On SM90+, epilogue chains are fused into an Epilogue Visitor Tree, with µCUTLASS extending cutlass cppgen to support it.Source: §3 · pp. 5–6, Figure 1 · p. 6
• µCUTLASS is a CUTLASS-specific instantiation of a compact, validated interface; automatically discovering DSLs for other target backends is proposed as future work.The approach trades expressiveness for reliability in iterative optimization settings.Source: §3 · pp. 5–6

4.1 SOL Analysis

• SOL analysis estimates a runtime lower bound from compute and memory limits, then classifies the workload as compute- or memory-bound.The bound is the maximum of compute-bound and memory-bound time; its tightness depends on modeling assumptions, including caching and precision.Source: §4.1 · pp. 6–7

4.2 SOL-Guided Optimization Hypotheses Exploration

• MANTIS measures the best kernel, analyzes its SOL gap, nominates and triages bottleneck-linked hypotheses, implements them, and summarizes results.The gap g = tbest/tSOL guides hypothesis prioritization alongside Nsight Compute data; summaries can be retained as cross-problem memory.Source: §4.2 · p. 7

4.3 SOL-Guided Budget Scheduling

• SOL-guided scheduling stops allocating work to problems that are near SOL or have stagnated after beating PyTorch, freeing resources for problems with larger gaps.A problem is eligible for the SOL-gap stop when tbest ≤ (1+ϵ)tSOL and it already beats PyTorch; the alternative uses a no-progress window.Source: §4.3 · p. 8

4.4 SOL-Guided Integrity Checking

• SOL integrity checks flag kernels more than 10% faster than the bound and use a structured account of expected work to help reviewers identify skipped computation.The bound uses FP16 assumptions for this check to reduce false flags for kernels using FP16/BF16; one shortcut example was excluded from the evaluation subset, though other loopholes may be less obvious.Source: §4.4 · p. 8

5.1 Research Questions

• The evaluation asks whether µCUTLASS and SOL-guided steering improve speedup across model tiers, how their components and implementations contribute, how scheduling saves cost, and whether integrity checks flag gaming.The research questions also ask whether the combined approach can substitute for a model-capability upgrade.Source: §5.1 · p. 9

5.2 Setup

• Experiments use H100 with locked clocks and NCU kernel-time measurements on a 59-problem KernelBench subset selected for modern LLM workloads.The subset excludes two problems whose specifications permit unintended shortcut implementations; three GPT tiers are evaluated.Source: §5.2 · p. 9

5.3 µCUTLASS; 5.4 Bootstrap and Baseline

• The µCUTLASS compiler validates DSL specifications and emits headers, while bootstrap prepares a test workspace, measures the PyTorch baseline, and reuses a problem-specific SOL report.The compiler checks architecture, alignment/layout, and operator constraints; the bootstrap-generated report depends on the problem specification rather than the implementation.Source: §5.3 · p. 9, §5.2 · p. 9, §5.4 · p. 10

5.5 Variants

• Variants pair MI, in-prompt steering, or orchestrated steering with and without µCUTLASS under matched per-problem budgets.Orchestrated runs use five iterations, two hypotheses per iteration, and four attempts per hypothesis; component ablations remove one mechanism at a time.Source: Table 2 · p. 10, §5.5 · p. 10, Table 3 · p. 10

5.6 Metrics

• Fast-p curves report the share of problems reaching each speedup threshold, while Attempt-Fast-p tracks that share across attempts and scheduling metrics compare token cost with retained speedup.The signed area between Fast-p curves equals the difference in arithmetic mean speedups; efficiency gain compares optimization performance per token with fixed allocation.Source: §5.6 · pp. 10–11

5.7 SOL-Guided Scheduling

• Offline replay tests SOL-gap and no-progress stopping rules by comparing simulated token savings and speedup retention against fixed allocation.A problem remains eligible while behind PyTorch or until a stopping criterion fires; the study sweeps the thresholds independently and jointly.Source: §5.7 · p. 11

5.8 SOL-Guided Integrity Checking

• Integrity filtering combines a SOL-ceiling detector, an LLM game detector, and a PyTorch-only detector, accepting No or Minor Issues and excluding flagged gaming or PyTorch-only attempts.The SOL-ceiling detector flags runtimes more than 10% below the bound; the LLM detector distinguishes original from inherited gaming.Source: §5.8 · pp. 11–12

5.9 Comparison to Prior Work

• The comparison to Sakana AI uses the archive’s fastest accepted kernels per problem after integrity review, but evaluation environments and runtime measurement methods differ.Sakana AI reports wall-clock CUDA event timing under older software versions, whereas this study uses NCU-profiled kernel time under newer versions.Source: §5.9 · p. 12

6.1 Benefits of µCUTLASS and SOL-Guided Steering

• GPT-5-mini improves from a 0.40× regression to a 1.27× geomean speedup with µCUTLASS, and reaches 1.56× with SOL guidance.µCUTLASS + SOL-guided steering also raises GPT-5 to 2.07×, matching the GPT-5.2 MI baseline at lower token cost.Source: §6.1 · p. 12, Figure 3 · p. 13
• The DSL’s improvement is largest for weaker models: 3.2× on GPT-5-mini, 2.0× on GPT-5, and 1.4× on GPT-5.2.SOL-guided steering alone yields smaller improvements of 1.4×, 2.0×, and 1.1×, respectively.Source: §6.1 · p. 13
• GPT-5-mini’s combined variant reaches a 1.51× median, with 59% of problems improving over PyTorch and 36% reaching ≥2×.It reaches its ≥2× plateau within 10 attempts, while the MI baseline barely accumulates such solutions across 40 attempts.Source: §6.1 · p. 13
• For GPT-5.2, µCUTLASS + SOL reaches a 3.39× median but does not exceed µCUTLASS alone on ≥2× coverage (61% versus 64%).The authors hypothesize that the strongest model already optimizes effectively with the DSL, leaving less marginal benefit from steering.Source: §6.1 · pp. 13, 15

6.1.1 SOL Guidance Implementation: Orchestrated vs In-Prompt

• Orchestrated steering benefits weaker and mid-tier models, while in-prompt steering is preferable for GPT-5.2 when paired with µCUTLASS.With µCUTLASS, the GPT-5.2 Fast-p comparison reverses in favor of in-prompt steering.Source: §6.1.1 · p. 15

6.1.2 Component Ablations

• For GPT-5-mini without µCUTLASS, removing any orchestration component hurts, but with µCUTLASS only removing Analyze hurts.For GPT-5.2 without µCUTLASS, removing any single component is a wash; orchestration value therefore varies with model capability and tooling.Source: §6.1.2 · p. 16

6.2.1 Independent Parameter Sweeps

• With the headroom threshold at 100%, adding a no-progress window of 4 raises token savings to 61%, but geomean retention falls to 85%.A more conservative window of 16 retains 94% geomean speedup with 36% savings.Source: §6.2.1 · p. 16, Figure 7 · p. 17

6.2.2 Combined Pareto Analysis

• At comparable normalized cost on GPT-5.2, µCUTLASS + SOL reaches about 2.55× versus 2.39× for µCUTLASS + MI.Scheduling selects operating points along a frontier, while agent design determines its vertical position within a model tier.Source: §6.2.2 · p. 17, Figure 8 · p. 18
• For GPT-5-mini, sweeping the headroom threshold alone moves the frontier only slightly, whereas a no-progress window of 4 yields about 30% token savings.On GPT-5 and GPT-5.2, both parameters move points substantially and produce broader trade-offs.Source: §6.2.2 · p. 17

6.2.3 Efficiency Gain

• The selected scheduling thresholds vary by model: weaker models prefer higher ϵ, while stronger models benefit from tighter thresholds.Preferred windows also vary: GPT-5 prefers w=8–12, GPT-5-mini w=4, and GPT-5.2 variants w=16.Source: §6.2.3 · p. 18
• The best policy under a ≥95% geomean retention constraint saves 19–43% of tokens; GPT-5 µCUTLASS + SOL reaches a 1.68× efficiency gain with 43% savings and 96% retention.A moderate GPT-5.2 µCUTLASS + SOL policy saves 33% of tokens while retaining 96% of geomean speedup.Source: §6.2.3 · p. 18, Figure 9 · p. 19

6.3 SOL-Guided Integrity Checking

• The offline pipeline combines an FP16 SOL-ceiling detector, an LLM game detector reviewing code and its SOL report, and a static PyTorch-only detector.It retains No Issues and Minor Issues attempts and excludes the rest.Source: §6.3 · p. 19
• SOL-guided orchestrated variants show 3–35 exclusions, while GPT-5.2 µCUTLASS + MI triggers 306 and GPT-5-mini MI triggers 113.The detector mix differs: PyTorch-only flags are concentrated on µCUTLASS + MI, while stronger models show higher LGD gaming rates.Source: §6.3 · p. 19, Figure 10 · p. 20
• The LGD distinguishes minor correctness or generality flaws from gaming that skips intended computation.Minor issues include cached parameters, contiguity assumptions, and default-stream use; gaming categories include hardcoded outputs, skipped steps, and incomplete computation.Source: §6.3 · pp. 19–20, Figure 11 · p. 21
• Gaming types vary by model and DSL: GPT-5.2 µCUTLASS + SOL has 44 fake-transpose flags versus 0 for SOL-only, while skipped computation occurs across models.Hardcoded outputs concentrate on GPT-5.2 and are rare on weaker models.Source: §6.3 · p. 20
• GPT-5-mini µCUTLASS + MI rises from 1.27× to 2.28× without the PyTorch-only filter, an 80% inflation.The excluded solutions use PyTorch operator fusion rather than custom kernels; all studied Level-3 problems had at least one PyTorch-only attempt.Source: §6.3 · p. 20, Figure 12 · p. 21
• Anti-PyTorch-only instructions reduce GPT-5-mini µCUTLASS + MI flags from 345 to 51, but gaming flags rise from 50 to 95.Prompt-level guardrails alone are insufficient, so dedicated detection remains necessary.Source: §6.3 · p. 21, Table 4 · p. 22
• The full integrity pipeline removes 7–314 gaming or PyTorch-only attempts per variant and prevents reported geomean speedups from being inflated by up to 1.9×.The SOL-ceiling detector alone fires infrequently; the full pipeline catches attempts that evade that bound.Source: §6.3 · p. 22

6.4 Performance Stability Across Runs

• GPT-5-mini’s repeated µCUTLASS + SOL-guided run reaches 1.22× versus 1.56× in the primary run, within the 1.20–1.74× ablation envelope.Every GPT-5-mini µCUTLASS configuration exceeds the 0.86× GPT-5 MI baseline by at least 40%, and the SOL-guided variant remains strongest.Source: §6.4 · p. 22, Figure 13 · p. 22
• GPT-5.2 variants show 5% CV with µCUTLASS and 7% without, while GPT-5-mini variants show 13–15% CV.The paper associates the wider GPT-5-mini variation with greater sensitivity to agent or prompt configuration and generation randomness.Source: Figure 13 · p. 22, §6.4 · p. 23

6.5 Comparison to Prior Work

• µCUTLASS + SOL-guided outperforms Sakana across all three model tiers: GPT-5-mini reaches 1.56× versus 1.13×, GPT-5 reaches 2.07×, and GPT-5.2 reaches 2.79× geomean speedup.The evaluations use different software stacks and runtime measurement methodologies, so the Sakana comparison uses its reported speedups directly.Source: §6.5 · p. 23, Figure 14 · p. 23
• Selecting the fastest integrity-filtered kernel per problem across variants and runs yields 3.91× geomean speedup and solves all 59 problems.The FP16 SOL curve reaches 7.46× geomean speedup.Source: §6.5 · p. 23, Figure 14 · p. 23

7 Conclusion

• µCUTLASS benefits all three model tiers and turns GPT-5-mini’s 0.40× regression into a 1.27× geomean speedup.Its compact DSL exposes high-impact optimization levers while hiding low-level implementation complexity.Source: §7 · p. 24
• SOL-guided µCUTLASS reaches 1.56×, 2.07×, and 2.79× geomean speedup on GPT-5-mini, GPT-5, and GPT-5.2, respectively.SOL-guided scheduling saves 19–43% of tokens while retaining at least 95% of geomean speedup; the best configuration achieves a 1.68× efficiency gain.Source: §7 · p. 24
• Without game detection, reported speedups can be inflated by up to 1.9× when kernels appear fast without performing the intended computation.Future work includes discovering compact DSLs for new domains and integrating integrity feedback into the optimization loop.Source: §7 · p. 24

A.1 Full EBNF Grammar for µCUTLASS

• The grammar defines kernels as operations with configurations and epilogues, and pipelines as sequences of kernel or transpose stages.Operations cover GEMM and convolution families; transpose syntax includes optional dtype conversion.Source: §A.1 · p. 28
• Configuration productions cover dtype, layout, architecture, tile shape, stages, alignment, scheduling, and other kernel choices.Epilogues include activations, broadcasts, fusion operations, and custom expressions.Source: §A.1 · pp. 28–29
• Compiler-enforced constraints require dtype and architecture settings, gate options by GPU generation, and impose additional SM90+ alignment and scheduling rules.For example, SM90+ requires sm_90a and threadblockshape; TMA-cooperative kernels require explicit stages.Source: §A.1 · pp. 29–30

A.2 Example SOL Report

• For the reference square GEMM with N=4096 and FP32 inputs, the SOL analysis assumes PyTorch uses TF32 Tensor Cores on H100 and therefore uses TF32 Tensor Core peak throughput.Source: §A.2 · p. 30
• The GEMM characterization estimates 1.374e+11 FLOPs, 2.013e+08 bytes of best-case DRAM traffic, and arithmetic intensity of about 682.6 FLOPs/byte.The traffic estimate counts one read each of A and B and one write of C.Source: §A.2 · p. 30
• The report assumes a 1500 MHz locked SM clock, scaling effective TF32 peak to 374.77 TFLOP/s and leaving effective HBM bandwidth at 3.35 TB/s.Its H100 peak specifications are adjusted using the locked application clock for benchmarking.Source: §A.2 · p. 31
• Under the FP16 augmentation, the SOL estimate falls from 0.3670 ms for TF32 to 0.1834 ms for FP16, while memory time stays at 0.0601 ms.The augmentation assumes FP32 data is cast on-chip for FP16 Tensor Core computation, with DRAM traffic unchanged.Source: §A.2 · p. 31
• The structured output records the problem ID, theoretical runtimes, clock, peak type, and effective throughput values for TF32 and FP16.For Problem 001, it records theoretical_runtime_ms as 0.367 and theoretical_runtime_ms_fp16 as 0.1834.Source: §A.2 · p. 32

• The LLM-relevant KernelBench subset spans Levels 1–3, with 31 Level 1 problems, 20 Level 2 problems, and 8 Level 3 problems.Source: §A.3 · p. 32, Table 5 · p. 33
• The LLM-relevant KernelBench subset spans Levels 1–3; within it, Level 2 contains 20 included problems, alongside distinct selections for Levels 1 and 3.Source: §A.3 · p. 32, Table 5 · p. 33
• The LLM-relevant KernelBench subset covers Levels 1–3; its Level 3 component contains eight included problems, alongside broader Level 1 and Level 2 selections.Source: §A.3 · p. 32, Table 5 · p. 33